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PREFACE 


This  document  is  the  final  technical  report  (CDRL  A003)  for  the  Factors  in 
Software  Quality  Study,  contract  number  F030602-76-C-0417.,  The  contract  was 
performed  in  support  of  the  U.S.  Air  Force  Electronic  Systms  Division's 
(ESD)  and  Rome  Air  Development  Center's  (RADC)  mission  t9 provide  standards 
and  technical  guidance  to  software  acquisition  marag^. 

The  report  was  prepared  by  J.  McCalJ^^r'ftichards,  and  6.  Walters  of  the 
Sunnyvale  Operations^^JjifornBrETonSystems  Programs,  General  Electric 
Company.  SignijEic^nt  contributions  were  made  by  A.  Breda,  S.  Reiss,  and 
R.  Colenso^/^ 

Technical  guidance  was  provided  by  J.  Cavano,  RADC  Project  Engineer  and 
Caf^tain  A.  French,  ESD  Technical  Monitor. 

The  report  consists  of  three  voliones,  as  followsj 

Concept  and  Definitions  of  Software  Quality 
Metric  Data  Collection  and  Validation^ 

Preliminary  Handbook  on  Software  Quality  for  an^ 

Acquisition  Manager, 

The  objective  of  the  study  was  to  establish  a concept  of  software  quality 
and  provide  an  Air  Force  acquisition  manager  with  a mechanism  to  quantita- 
tively specify  and  measure  the  desired  level  of  quality  in  a software 
product.  Software  metrics  provide  the  mechanism  for  the  quantitative  specifi- 
cation and  measurement  of  quality. 

This  first  volume  describes  the  process  of  developing  our  concept  of 
software  quality  and  what  the  underlying  software  attributes  are  that 
provide  the  quality,  and  defines  the  metrics  which  provide  a measure  of 
the  degree  to  which  the  attributes  exist. 
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The  Air  Force  is  constantly  striving  to  improve  the  quality  of  its 
software  systems.  Producing  high  quality  software  is  a pferequislte  for 
satisfying  the  stringent  reliability  and  error- f»«e  requirements  of  com- 
mand and  control  software.  To  help  accomplish  this,  a more  precise  def- 
inition of  software  quality  is  needed  as  well  as  a way  to  derive  metrics 
for  quantifying  software  for  objective  analysis.  This  effort  was  initi- 
ated in  response  to  the  need  to  better  understand  those  factors  affecting 
software  quality  and  fits  into  the  goals  of  RADC  TPO  No.  5,  Software  Cost 
Reduction  in  the  area  of  Software  Quality  (Metrics).  General  Electric 
classified  over  the  complete  range  of  software  development  both  user- 
oriented  and  software-oriented  characteristics  which  were  related  to  Air 
Force  applications  and  life-cycle  phases.  Programming- language  independ- 
ent metrics  were  defined  using  Air  Force  data  bases.  Finally,  formal 
methodology  for  the  validation  of  the  metrics  was  developed  and  used. 


The  significance  of  this  work  is  that  through  the  establishment  of 
quality  measurement  a beneficial  impact  will  occur  on  the  evaluation  and 
implementation  of  a software  product  at  each  stage  of  development.  Trade- 
offs between  technical  value  and  cost  will  be  more  easily  understood.  In 
addition.  Air  Force  acquisition  managers,  with  the  aid  of  a handbook  de- 
livered as  part  of  this  contract,  will  be  able  to  specify  requirements  to 
software  developers  more  completely  and  then  determine  whether  those  re- 
quirements are  being  satisfied  early  enough  for  corrective  action.  As 
quality  measurement  becomes  more  vigorous  in  the  future,  the  Air  Force  will 
be  capable  of  establishing  software  product  and  service  standards  for 
itself  and  its  contractors. 


f- 

JOSEPHS.  CAVANO 
Project  Engineer 
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SECTION  1 

INTRODUCTION/EXECUTIVE  SUMMARY 

1.1  TASK  OVERVIEW 

The  Factors  In  Software  Quality  task  was  conducted  In  support  of  the 
U.S.  Air  Force  Electronic  Systems  Division's  (ESD)  and  Rome  Air  Development 
Center's  (RAOC)  mission  to  provide  standards  and  technical  guidance  to  soft* 
ware  acquisition  managers.  ESD  sponsored  the  task  and  RAOC  provided 
technical  project  management. 

The  impetus  for  this  effort  and  other  related  work  in  the  analysis  of  soft- 
ware quality  can  be  traced  to  recommendations  for  such  research  made  jointly 
by  DOD,  industry,  and  university  .’^presentati ves  at  the  Symposium  on  the  High 
Cost  of  Software  [wULFWTs]  in  September,  1973,  at  the  Joint  Logistics  Commanders 
Electronic  Systems  Reliability  Workshop  (by  members  of  the  Software  Reliability 
Working  Group)  [fINDTS]  in  May,  1975,  and  more  recently  by  the  DOD  R&D  Panel. 

1.2  TASK  OBJECTIVES 

In  the  acquisition  of  a new  software  system,  a major  problem  facing  a System 
Program  Office  (SPO)  is  to  specify  the  requirements  to  the  software  developer, 
and  then  to  determine  whether  those  requirements  are  being  satisfied  as  the 
software  system  evolves.  The  parameters  of  the  specification  center  about  the 
technical  definition  of  the  application  and  the  software  role  within  the  over- 
all system.  Following  this,  a realistic  schedule  and  costs  are  negotiated. 

While  the  application  functions,  cost,  and  schedule  aspects  of  development 
can  be  objectively  defined,  measured,  and  assessed  throughout  the  development 
of  the  system,  the  quality  desired  has  historically  been  definable  only  in 
subjective  terms.  This  occurs  because  the  SPO  has  no  quantifiable  criteria 
against  which  to  judge  the  quality  of  the  software  until  he  begins  to  use  the 
system  under  operational  conditions. 
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As  represented  In  Figure  1.2-1,  the  objective  of  this  study  was  to  provide 
guidelines  In  how  to  objectively  specify  the  desired  amount  of  quality  at 
the  system  requirements  specification  phase.  By  levying  measurable  quality 
criteria  on  the  developer,  the  SPO  will  be  able  to  subsequently  evaluate  the 
quality  of  the  software  not  only  when  the  system  becomes  operational,  but 
also  as  each  phase  of  the  projKvt  Is  completed.  As  a result  of  corrective 
actions  the  SPO  may  choose  to  Invoke,  these  early  measurements  can  signifi- 
cantly reduce  Impact  on  life  cycle  cost  and  schedule. 

The  figures  drawn  with  solid  lines  represent  the  questions  the  SPO  can  now  ask 
and  can  obtain  objective  answers.  The  figures  drawn  with  dashed  lines  repre- 
sent areas  which  cannot  presently  be  addressed.  The  objective  of  this  task  was 
to  provide  the  mechanism  to  answer  the  question  of  how  good  the  software  Is 
more  precisely  and  earlier  In  the  life  cycle.  The  results  of  this  task 
provide  the  basis  for  the  SPO  to  specify  and  evaluate  the  software  quality 
quantitatively,  as  Is  Illustrated  with  the  dash-lined  figures. 

The  approach  taken  to  quantify  software  quality  Is  summarized  as  follows: 

1. '  Determine  a set  of  quality  factors  which  jointly  comprise  software 

quality.  (Section  2,3) 

2.  Develop  a working,  hierarchical  definition  by  Identifying  a set  of 
criteria  for  each  factor.  (Section  4) 

3.  Define  metrics  for  each  criterion  and  a normalization  function  which 
relates  and  Integrates  the  metrics  for  all  of  the  criteria  of  a 
factor  to  an  overall  rating  of  that  factor.  A scaling  of  the  metrics' 
contributions  to  this  rating  will  result  In  a figure  of  merit  for 
each  factor.  (Section  5,6,7) 

4.  Validate  the  metrics  and  normalization  functions  by  utilizing  the 
historical  data  of  two  Air  Force  systems.  (Section  7,8) 

5.  Translate  the  results  of  this  effort  Into  guidelines  that  can  be  used 
by  Air  Force  Program  Offices  to  specify  the  quality  of  the  software 
product  required  and  to  measure  to  determine  If  the  development  effort 
Is  leading  toward  that  level  of  quality.  (Volume  III) 
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Figure  1.2-1  Specifying  and  Measuring  Quality  Software 


In  taking  this  approach,  we  established  a comprehensive  framework  which 
facilitates  the  Incorporation  of  future  efforts  and  refinements  to  thl^ 
metrics  and  their  correlation  to  the  quality  factors.  Also,  recoriinendatlons 
are  made  on  hoii  the  metrics  should  be  collected. 

The  results  of  this  task  provide  the  SPO  with  a Mlithodology  for  specify- 
ing the  quality  he  wants  In  the  software  and  the  pmcndiires  for  determining 
If  he  Is  realizing  the  level  of  quality  that  was  specified.  By  achieving 
this  goal,  the  SPO  will  have  objective  Insight  Into  the  software  quality 
throughout  the  acquisition  process. 

1.3  ACKNOWLEDGMENT  OF  PREVIOUS  WORK 
In  establishing  the  framework  for  this  study  of  factors  In  software  quality, 
we  are  attempting  to  Incorporate  the  work  that  others  have  done  previously 
In  this  area.  An  extensive  literature  search  was  conducted.  The  references 
are  listed  following  the  Appendices  and  the  major  references  are  abstracted 
In  the  bibliography. 

We  used  other  RADC  sponsored  efforts,  particularly  those  In  the  area  of 
reliability  and  maintainability,  as  Input  to  this  task.  The  planned  approach 
was  to  concentrate  In  those  areas  where  little  work  has  been  done. 

1.4  CONTRIBUTION  TO  STATE  OF  KNOWLEDGE 

This  study  has  been  directed  at  expanding  upon  the  current  state  of  knowledge 
about  software  quality.  The  following  aspects  of  our  approach  are  Identified 
as  expansions  to  the  work  to  date: 


• Provide  a global  view  of  software  quality  - most  previous  efforts  have 
evaluated  subsets. 

e Provide  a formal  methodology  for  the  validation  of  metrics. 

• Relate  the  quality  factors  to  Air  Force  applications. 

• Relate  the  quality  factors  to  the  life  cycle  phases. 
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• Define  metrics  which  are  programming  language  Independent. 

• Identify  metrics  which  can  be  applied  early  1n  the  development 
phase  (during  requirements  analysis  and  design). 

• Attempt  to  choose  criteria  that  are  as  Independent  and  unambiguous 
as  possible. 

• Attempt  to  quantify  the  correlation  of  subjective  criteria  to  the 
quality  factors. 

• Identify  automated  metric  data  collection  tools. 

• Provide  a framework  for  factors  in  software  quality  that  can  be  used 
in  future  research  efforts. 

1.5  CONCLUSIONS  OF  THE  STUDY 

The  effort  represents  a conceptual  study  investigating  the  factors  of  soft- 
ware quality.  Our  intent  was  to  build  upon  the  significant  contributions 
of  other  efforts  in  recent  years  related  to  understanding  software  quality. 

The  main  thrusts  of  this  study  were  the  formulation  of  an  SPO-oriented  concept 
of  factors  In  software  quality  and  the  establishtr»nt  and  application  of 
metrics  oriented  toward  the  early  phases  of  development.  The  measures  are 
Indicators  of  the  progress  toward  the  desired  quality.  They^lso  give 
an  early  Indication  of  the  quality  factors  that  are  not  realiz^d-jn  testing 
or  during  Initial  operation  but  have  a large  cost  impact  later  in  thV.l^fe 
of  a product,  e.g.,  portability,  reusability,  or  interoperability. 

The  complete  procedure  of  establishing  a framework  for  factors  in  software 
quality,  defining  the  factors,  relating  them  to  Air  Force  applications  and 
the  life-cycle  phases,  establishing  criteria,  defining  them,  using  them 
to  identify  the  relationships  and  tradeoffs  between  factors,  defining  metrics, 
establishing  their  relationship  to  the  quality  factors,  and  validating  the 
relationship  was  an  iterative  process.  It  has  been  described  in  this 
report  in  a sequential  manner  only  for  clarity  and  simplicity.  It  will 
continue  to  evolve  as  more  experience  is  gained  through  the  application  of 
metrics  to  more  software  developments. 
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The  framework  established  Is  flexible  and  expandable.  It  provides  a complete 
view  of  software  quality.  It  provides  a mechanism  for  specifying  and  measur- 
ing the  quality  of  a software  product.  The  following  benefits  can  be 
realized  from  this  conceptualization  of  factors  In  software  quality: 

e It  Is  a simple,  comprehensive  tool  for  an  acquisition  manager  to  use  - 
guidelines  for  Its  use  are  provided  In  the  form  of  a handbook. 

(Volume  III) 

e It  provides  the  acquisition  manager  with  a life-cycle  view  of  his 
software  product,  forcing  consideration  of  such  factors  as  main- 
tainability and  portability  In  the  system  specification  phase, 
e It  provides  a mechanism  for  performing  high-level  tradeoff  studies 
early  In  the  life  cycle  (requirements  analysis,  performance  require- 
ments analysis,  and  preliminary  design)  to  help  In  determining  the 
product's  required  capabilities  and  performance  characteristics, 
a as  the  software  development  process  technology  advances  and  new 
development  techniques  are  Introduced,  the  metrics  can  easily  and 
logically  be  modified  or  added. 

The  set  of  metrics  established  provides  a comprehensive  coverage  of  the  char- 
acteristics of  a software  product.  As  they  exist,  they  represent  an  excellent 
guideline  for  testers,  quality  assurance  personnel,  and  Independent  verifi- 
cation and  validation  efforts.  They  also  incorporate  an  extensive  composite 
of  a number  of  texts  on  good  programming  practices  and  style. 


The  specific  results  of  the  validation  phase  of  the  study  allow  the  conclusion 
that  software  metrics  are  a viable  concept.  The  regression  analysis  showed 
significant  correlation  for  some  metrics  with  related  quality  factors. 
Quantitative  metrics  can  be  applied  to  Intermediate  products  of  the  soft- 
ware development  which  exist  as  early  as  the  requirement  analysis.  As 
more  disciplined,  software  engineering  approaches  are  taken  toward  the 
development  of  software,  the  more  applicable  quantitative  metrics  become. 


The  establishment  of  generalized  precise  normalization  functions  was  beyond 
the  scope  of  the  study.  The  limiting  factors  were  that  the  sample  of  mod- 
ules and  systems  was  not  large  enough,  general  enough,  nor  had  the  two 
systems,  which  were  used,  been  through  all  of  the  quality  factors  related 
activities  (e.g. , moved  to  another  environment,  linked  to  another  system, 
etc.}.  The  sample  was  representative  of  two  large-scale  developments  so 
I the  experience  of  applying  the  metrics  contributed  considerable  knowledge 

to  the  software  quality  technology.  One  other  limiting  factor  was  that 
the  measures  were  biased  high  because  the  metrics  were  applied  after  the 
two  systems  had  been  delivered.  So,  even  though  the  metrics  were  applied 
to  software  products  delivered  during  the  development  they  had  been  updated 
to  reflect  all  of  the  changes  and  fixes  made  to  the  system  as  a result  of 
testing  and  operational  experience.  A definite  recommendation  of  this 
study  then  Is  to  apply  the  metrics  during  the  actual  development  of  a soft- 
ware system  to  further  validate  their  relationship  to  the  resulting  quality. 

In  deriving  the  set  of  metrics,  the  number  of  metrics  became  a significant 
consideration.  The  concept  of  applying  the  same  metric  successively 
during  the  development  phases  helped  contain  the  problem  of  an  unwieldy 
number  of  metrics.  The  fact  that  many  of  the  metrics  can  be  collected 
i automatically  assists  In  making  the  present  set  more  manageable, 

i 

l 

? 

A large  number  of  existing  software  support  tools  were  Identified  that  pro- 
vide metric  data  collection  capabilities.  Significantly,  several  tools 
were  Identified,  and  some  applied,  which  automate  the  collection  of  metrics 
in  the  requirements  analysis  and  design  phases  of  the  development.  Several 
other  tools  can  be  developed.  Because  many  tools  do  exist  that  provide  a 
subset  of  the  overall  capabilities  of  data  collection  required,  an  Integrated 
approach  must  be  developed  to  effectively  collect  metric  data  In  any  soft- 


e ' ware  development  environment. 


Some  very  practical,  beneficial  results  from  the  application  of  the  metrics  In 
their  current  form  have  been  Identified.  When  the  metrics  are  applied  to  the 
set  of  software  products  available  at  various  times  during  the  development, 
they  can  be  used  as  Indicators.  Low  measurements  Identify  modules  or  charac- 
teristics which  should  be  Investigated  and  the  scores  Justified.  The  meth- 
odology for  regression  analysis  described  can  be  used  In  conjunction  with 
this  metric  Indicator  concept.  The  analysis  provides  an  Indication  of  what 
specific  software  characteristics  vary  In  a particular  environment  relative 
to  variations  In  software  quality,  I.e.,  which  characteristics  vary  signi- 
ficantly and  cause  variation  In  the  software  quality. 

This  Information  Is  beneficial  to  software  developers  In  writing  their 
design  and  programming  standards  and  conventions.  It  Is  also  beneficial 
to  QA  personnel  In  Identifying  areas  or  modules  requiring  attention  during 
development  and  concentrated  testing. 

An  SPO  can  use  the  quantitative  nature  of  the  metrics  and  the  framework 
of  the  software  quality  factors  to  specify  the  required  level  of  software 
quality  quantitatively.  By  specifying  the  software  quality  In  terms  of 
the  metrics,  the  SPO  Is  specifying  the  desired  characteristics  of  the 
software.  The  characteristics  are  essentially  Independent  of  method  or 
phllosoply  of  software  development  so  there  are  no  unjustified  restrictions 
placed  on  the  software  developer. 

The  software  quality  metrics  represent  the  Introduction  of  a more  disci- 
plined engineering  approach  to  software  quality  assurance.  They  provide 
a quantitative  tool  for  the  Inspection  and  evaluation  of  software  products 
during  the  development  phase. 

1.6  FURTHER  RESEARCH 

Several  areas  for  further  research  were  Identified  during  this  effort. 
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The  exercise  of  determining  the  set  of  metrics  revealed  several  areas  requiring 
further  Investigation.  Within  the  transition  phase,  the  two  quality  factors, 
reusability  and  Interoperability,  are  relatively  untouched  In  the  literature, 
tittle  research  has  been  conducted  to  determine  what  constitutes  reusability 
and  Interoperability  or  what  software  attributes  provide  these  qualities. 

It  Is  felt  that  further  research  In  these  areas  could  have  potentially  high 
life-cycle  cost  benefits. 


A second  area  where  we  feel  further  research  would  be  beneficial  Is  In 
measuring  various  aspects  of  efficiency.  Because  many  of  the  attributes  of 
efficiency  have  a negative  effect  on  all  other  quality  factors.  It  Is  an 
Important  consideration  of  the  software  quality  concept.  Most  current 
measures  of  efficiency  are  dynamic  measures  requiring  execution  of  the 
code.  In  deriving  some  static  measures  we  realized  that  an  Integrated 
set  of  both  dynamic  and  static  measures  are  necessary  to  judge  the  degree 
of  efficiency.  Further  work  Is  required  to  develop  this  type  of  measure. 

Further  research,  application,  and  experience  are  required  to  formalize 
the  normalization  functions.  This  report  has  stressed  the  methodology 
of  deriving  and  validating  the  mrmalization  functions  to  encourage  the 
application  of  these  techniques  to  other  software  developments.  Use  on 
future  developments  will  add  to  the  data  base  for  the  establishment  of 
generalized  normalization  functions,  as  well  as  provide  Indication  to  the 
SPO  and  software  developer  of  their  progression  toward  a high  quality 
product.  It  will  also  contribute  to  the  error  data  collection  technology 
and  experience. 

As  previously  mentioned,  the  metrics  should  be  applied  during  a software 
development  to  obtain  more  realistic  measures.  It  Is  also  recommended 
that  the  metrics  be  applied  to  specific  projects  Involving  (1)  software 
conversions  from  one  environment  to  another  to  validate  the  metrics 
related  to  portability,  (2)  efforts  linking  two  systems  to  validate  the 
Interoperability  metrics,  and  (3)  efforts  upgrading  a system  to  validate 
the  reusability  metrics.  These  efforts  would  not  only  provide  a chance 
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for  validation  of  the  particular  metrics  but  also  give  considerable  Insight 
Into  additional  metrics  In  these  high-payoff*  late-llfe-cycle-impact 
quality  factors. 
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SECTION  2 


DETERMINATION  OF  QUALITY  FACTORS 
2.1  DEFINITION  OF  TERMS 

To  be  consistent  In  our  determination  of  factors,  criteria,  and  metrics,  we 
first  established  a set  of  working  definitions.  This  was  done  In  order  to 
provide  a framework  from  which  to  more  objectively  Judge  candidate  quality 
factors.  The  working  definitions  are  as  follows: 

• Software;  the  programs  and  documentation  associated  with  and  result- 
ing from  the  software  development  process. 

• Quality:  a general  term  applicable  to  any  trait  or  characteristic, 
whether  Individual  or  generic,  a distinguishing  attribute  which  Indi- 
cates a degree  of  excellence  or  Identifies  the  basic  nature  of 
something. 

• Factor;  a condition  or  characteristic  which  actively  contributes  to 
the  quality  of  the  software.  For  standardization  purposes,  all  factors 
will  be  related  to  a normalized  cost  to  either  perform  the  activity 
characterized  by  the  factor  or  to  operate  with  that  degree  of  quality. 
For  example,  maintainability  Is  the  effort  required  to  locate  and 

fix  an  error  In  an  operational  program.  This  effort  required  may  be 
expressed  In  units  'uch  as  time,  dollars,  or  manpower.  The  following 
rules  were  used  to  determine  the  prime  set  of  quality  factors: 

- a condition  or  characteristic  which  contributes  to  software  quality, 

- a user-related  characteristic, 

- related  to  cost  either  to  perform  the  activity  characterized  by 
the  function  or  to  operate  with  that  degree  of  quality, 

- relative  characteristic  between  software  products. 

The  last  rule,  that  a factor  Is  a relative  characteristic  between  software 
products,  requires  a brief  explanation.  Figure  2.1-1  Illustrates  the  relation- 
ship between  a factor  and  the  cost  to  achieve  different  levels  of  that  quality 
factor.  As  an  example,  we  will  assume  the  curve  describes  the  cost  to  level-of- 
quallty  relationship  for  the  factor,  reliability.  A much  lower  level  of  reli- 
ability, which  costs  less  to  achieve,  may  be  as  acceptable  to  a management 
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Figure  2.1-1  Relationship  of  Software  Quality  to  Cost 

Information  system  (MIS)  acquisition  manager  as  a much  higher  level  Is  to  a 
connand  and  control  (C^)  manager  due  to  the  nature  of  the  applications.  So, 
while  the  c2  final  product  may  have  a higher  degree  of  reliability  according 
to  our  measures.  It  Is  no  more  acceptable  to  Its  user  than  the  MIS  system  with 
Its  lower  reliability  Is  to  Its  user.  This  relationship  Is  further  Illustrated 
In  Section  3 where  the  quality  factors  are  related  to  specific  Air  Force 
applications. 

• Criteria:  attributes  of  the  software  or  software  production  process 
by  which  the  factors  can  be  judged  and  defined.  The  following  rules 
were  applied  to  the  determination  of  criteria: 

- attributes  of  the  software  or  software  products  of  the  development 
process:  I.e.,  criteria  are  software  oriented  while  factors  are 
user  oriented, 

- may  display  a hierarchical  relationship  with  subcriteria, 

- may  affect  more  than  one  factor. 

e Metrl cs : measures  of  the  criteria  or  subcriteria  related  to  the 
quality  factors.  The  measures  mqy  be  objective  or  subjective.  The 
units  of  the  metrics  are  chosen  as  the  ratio  of  actual  occurrences 
to  the  possible  number  of  occurrences.  Metrics  will  be  discussed 
further  In  Section  6. 
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2.2  IDENTIFICATION  OF  QUALITY  FACTORS  IN  THE  LITERATURE 

A literature  search  was  conducted  to  assemble  all  current  definitions  and  to 
Identify  any  applicable  discussions  with  respect  to  software  quality  factors. 
Table  2.2-i  sunmarlzes  the  list  of  terms  extracted  from  the  literature  and 
represents  the  baseline  of  potential  or  candidate  quality  factors  referenced 
In  this  study. 

This  list  of  approximately  55  terms  was  used  as  the  starting  point  for  deter- 
mining the  prime  set  of  factors.  The  next  task  was  to  apply  the  definitions 
given  In  Section  2.1  to  the  list  of  candidate  factors.  The  intent  of  this 
exercise  was  to  put  Into  place  a standard  by  which  to  judge  terms  with  regard 
to  consistency,  redundancy,  suitability,  etc.  The  results  of  applying  the 
definitions  to  the  candidate  terms  is  discussed  in  further  detail  below,  where 
the  rationale  for  terms  such  as  understandablllty,  modularity,  and  complexity 
is  explained. 

In  Table  2.2-2  we  provide  a brief  cross-reference  of  definitions  and  authors 
quoted.  The  total  set  of  definitions  analyzed  in  this  report  appears  in 
Appendix  A where  work  by  various  researchers  in  the  software  community  are 
quoted  or  paraphased. 

2.3  THE  PROCESS  OF  GROUPING  CANDIDATE  FACTORS 


The  list  of  potential  factors  established  in  Table  2.2-1  was  known  to  contain 
obvious  redundancy  and  some  terms  which  do  not  comply  with  all  of  the  rules 
identified  for  the  prime  set  of  factors.  It  was  also  felt  that  the  list  was 
far  too  long  to  represent  a manageable  set  of  factors.  For  this  reason,  some 
guidelines  were  generated  to  aid  in  grouping  the  factors  into  a smaller,  con- 
cise number  of  entries  which  still  cover  the  comprehensive  set  of  software 
quality  factor  characteristics  desired.  The  guidelines  used  were: 

e User-oriented  terms  are  potential  factors;  software-oriented  terms 
are  potential  criteria. 

e Synonyms  that  are  Identified  are  grouped  together. 


Idble  2.2-1  Candidate  Software  Quality  Factors 
Extracted  from  the  Literature 


PORTABILITY 

AUGMENTABILITY 

TRANSFERABILin 

INTEGRITY 

ACCEPTABILITY 

SECURin 

COMPLETENESS 

PRIVACY 

CONSISTENCY 

USABILITY 

CORRECTNESS 

OPERABILITY 

AVAILABILITY 

HUMAN  FACTORS 

RELIABILITY 

COMMUNICATIVENESS 

ACCURACY 

STRUCTUREDNESS 

ROBUSTNESS 

MODULARITY 

EFFICIENCY 

UNIFORMITY 

PERFORMANCE 

GENERALITY 

CONCISENESS 

REUSABILITY 

UNOERSTANOABILITY 

TESTABILITY 

SELF-DESCRIPTIVENESS 

INTEROPERABILITY 

CLARITY 

CONVERTIBILITY 

LEGIBILITY 

MANAGEABILin 

MAINTAINABILITY 

COST 

STABILITY 

ACCOUNTABILin 

ADAPTABILITY 

SELF-CONTAINEDNESS 

EXTENSIBILITY 

EXPRESSION 

MODIFIABILITY 

VALIDITY 

ACCESSIBILITY 

TIME 

FLEXIBILITY 

COMPLEXITY 

EXPANDABILITY 

PRECISION 

DOCUMENTATION 

TOLERANCE 

REPAIRABILITY 

COMPATABILITY 

SERVICEABILITY 
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• Logically  similar  terms  are  grouped  together. 

• Required  a manageable  number  of  groups  - the  prime  set  of  factors 
should  be  small  In  number  to  maintain  a simple  frameMork  In  which 
the  SPO  can  work  most  effectively. 

A central  Issue  In  the  grouping  process  was  to  identify  only  user-oriented  terms 
as  potential  factors.  This  has  the  advantage  of  defining  for  the  SPO  a set  of 
user-oriented  terms  for  specifying  the  relative  amount  of  quality  desired  In  the 
product.  It  also  enabled  us  to  significantly  reduce  the  number  of  potential 
factors.  In  defining  software-oriented  terms  as  criteria,  then,  we  also  could 
establish  more  realistic  standards  by  which  to  judge  the  final  software  product 
which  will  be.  In  turn,  easier  to  measure. 

2.4  RESULTS  AND  RATIONALE  AFTER  GROUPING  QUALITY  FACTORS 
Table  2.4-1  provides  the  results  of  the  grouping  process.  The  underlined  terms 
were  chosen  as  the  group  names.  They  were  chosen  because  they  were  the  most 
descriptive  or.  If  a hierarchical  relationship  existed  in  the  group,  the  higher 
member  was  chosen. 

Three  groupings  were  determined  not  to  be  candidates  for  a quality  factor; 
Understandablllty,  Complexity,  and  Modularity.  Complexity  and  modularity  are 
software-oriented  rather  than  user-oriented  terms.  The  user  is  Interested 
In  such  things  as  how  fast  the  program  runs  (efficiency)  and  how  easy  It  Is 
to  maintain  (maintainability),  not  how  modular  it  Is.  Modularity  is  not  an 
end  Item  quality  or  performance  characteristic.  Since  It  Is  software  oriented, 
it  contributes  to  several  of  the  candidate  quality  factors,  and  Is  therefore 
a candidate  for  a criterion.  Similarly,  complexity  Is  a candidate  for  a 
criterion  (simplicity  will  be  used  to  connote  a positive  attribute). 

Understandablllty  was  Initially  Identified  as  a quality  factor.  Upon  further 
analysis  (relating  factors  to  Air  Force  applications  and  to  life-cycle  phases), 
we  decided  that  It  was  not  quantifiable.  Measuring  how  well  software  Is  under- 
stood Is  extremely  difficult  and  associating  a cost  would  be  even  more  difficult. 
The  reasons  for  wanting  to  understand  a program  really  are  related  to  using, 
maintaining,  or  changing  the  program.  Thus,  the  quantifiable  attributes  of 
understandablllty  will  be  used  as  criteria  of  other  factors. 
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Table  2.4-1  Grouping  of  Software  Quality  Factors 
to  Achieve  Unambiguous  Set 


CORRECTNESS 

UNOERSTANDABILITY 

PORTABILITY 

Acceptabi 1 1 ty 

Clarity 

Transferability 

Completeness 

Legibility 

Compatablllty 

Consistency 

Self-0escr1pt1 veness 

Expression 

Val  i d1  ty 

Performance 

RELIABILITY 

MAINTAINABILITY 

REUSABILITY 

Availability 

Stability 

General 1 ty 

Accuracy 

Manageability 

Utility 

Robustness 

Conciseness 

Precision 

Repalrability 

Tolerance 

Serviceability 

EFFICIENCY 

FLEXIBILITY 

INTEROPERABILITY 

INTEGRITY 

Adaptability 

Extensibility 

Accessibility 

Expandability 

Augmentabllity 

Modifiability 

COMPLEXITY 

Security 

Pri vacy 

USABILITY 

TESTABILITY 

MODULARITY 

Operability 

Accountability 

Structuredness 

Human  Factors 

Uniformity 

Communicativeness 

Sel f-Contai nedness 

Convertibility 

DOCUMENTATION 

COST 

TIME 

Integrity,  security,  and  privacy  were  grouped  together.  Integrity  Is  Inter- 
preted as  the  ability  of  the  software  to  protect  against  unauthrolzed  access  - 
software  Integrity.  Another  coninon  Interpretation,  data  base  Integrity,  (the 
ability  of  the  software  to  maintain  an  accurate  data  base  In  a multiaccess 
environment)  falls  under  the  major  category  of  reliability.  Privacy  Is  the 
ability  to  control  the  use  of  data.  An  authorized  person  may  access  data  and 
then  use  It  In  an  unauthorized  manner.  To  date,  little  software  has  been 
developed  for  providing  the  capability  to  control  usage  of  data.  In  this 
respect,  privacy  Is  outside  the  scope  of  this  study.  We  will  maintain  It  as 
a part  of  the  quality  factor  called  Integrity  for  future  expansion. 

Two  other  groupings,  cost  and  time,  provide  the  baseline  for  evaluating  the 
factors.  It  costs  more  and  takes  more  time  to  develop  a more  reliable  system. 
A system  that  Is  not  flexible  costs  more  and  takes  longer  to  change.  Cost 
and  time,  therefore,  were  not  considered  candidates  for  quality  factors,  but 
form  the  basis  for  correlating  metrics  with  the  various  levels  of  quality. 
Documentation  Is  one  of  the  vehicles  for  providing  the  various  qualities. 
Specific  documents  enhance  the  maintainability,  others  the  testability, 
and  so  on.  Documentation  was  not  considered  a quality  factor  but  a product 
of  the  software  development  process  which  can  be  evaluated  to  obtain  a 
measure  of  quality. 
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I DEFINITIONS  OF  QUALITY  FACTORS 

3.1  CONCEPTUALIZATION  OF  FACTORS  IN  SOFTWARE  QUALITY 

Through  evaluation  and  analysis  of  the  groupings  of  factors,  three  different 
orientations  that  one  could  take  In  looking  at  a delivered  software  product 
were  recognized.  These  three  orientations  took  on  added  significance  because 
of  their  relevance  to  an  SPO.  The  SPO's  Interaction  with  a delivered  soft- 
ware product  can  be  described  In  terms  of  only  three  distinct  activities  as 
follows: 

Software  Product  Activities 

• Product  Operation 

• Product  Revision 

• Product  Transition 

This  scheme  was  adopted  as  the  framework  about  which  we  would  further  evolve 
our  software  quality  research.  Figure  3.1-1  illustrates  the  concept  derived. 
The  questions  In  parentheses  provide  the  relevancy  or  interpretation  of  the 
factors  to  an  SPO. 

To  date,  the  major  emphasis  has  always  been  on  Initial  product  operation. 
Specifications  and  testing  only  sti«ss  factors  such  as  the  correctness  or 
reliability.  There  Is,  however,  a growing  recognition  of  the  longer-term 
Implications  In  our  software  development,  such  as  the  flexibility  or  main- 
tainability of  the  software  product.  Analyses  of  life-cycle  costs  have  shown 
that  the  costs  of  maintenance  and  redesign  exceed  the  cost  of  Initial  develop- 
ment [lIEBEZz]  and  that  the  cost  of  fixing  errors  after  a system  is  operational 
Is  up  to  30  times  greater  than  If  the  error  was  caught  during  system  testing. 
Several  software  developers  state  as  policy  a rule  of  thunb  that  If  a mod- 
ification requires  XX  of  the  lines  of  code  to  be  revised,  a total  redesign  Is 
undertaken;  X varies  between  20  and  50  percent.  All  of  these  facts  point  to 
the  conclusion  that  our  software  systems  are  not  designed  and  developed  with 
the  required  emphasis  on  maintainability,  flexibility,  portability,  etc. 
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significant  Impacts  to  the  total  cost  of  a system  during  the  life  of  the 
system  can  be  realized  because  of  the  following  reasons: 
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a maintenance  and  operations  costs  are  very  high 

• major  program  modifications  are  required 

• there  Is  a change  of  mission 

e a change  occurs  In  the  hardware 

• there  Is  a change  of  users,  either  a new  Air  Force  user  or 
new  contractor 

An  occurrence  of  any  of  the  above  events  usually  requires  redesign,  recoding, 
and  retesting.  The  size  of  the  Impact  of  one  of  these  events  on  total  system 
cost  Is  a function  of  the  quality  factors  associated.  In  particular,  with 
product  revision  and  product  transition.  In  order  to  minimize  the  Impact, 
these  factors  must  be  taken  Into  account  1n  the  Initial  program  development. 

The  Air  Force  Systems  Command,  in  order  to  provide  more  of  a focus  on  these 
problems,  has  initiated  efforts  to  Improve  their  total  life-cycle  management 
techniques  [aIRF76].  The  Naval  Sea  Systems  Command  has  Initiated  the  PATHWAY 
Program,  In  which  they  recognize  the  Importance  of  portability  and  reusability 
In  a software  product  [PATH76]. 

This  conceptualization  of  factors  in  software  quality  provides  a mechanism  for 
the  SPO  to  quantify  these  concerns  for  the  longer  life-cycle  Implications  of 
the  software  product.  For  example.  If  the  SPO  Is  sponsoring  the  development 
of  a system  In  an  environment  In  which  there  Is  a high  rate  of  technical  break- 
throughs In  hardware  design,  portability  should  take  on  an  added  significance. 
If  the  expected  life  cycle  of  the  system  Is  long,  maintainability  becomes  a 
cost-critical  consideration.  If  the  system  Is  an  experimental  system  where 
the  software  specifications  will  have  a high  rate  of  change,  flexibility  In 
the  software  product  Is  highly  desirable.  If  the  functions  of  the  system  are 
expected  to  be  required  for  a long  time,  while  the  system  Itself  may  change 
considerably  from  time  to  time,  reusability  Is  of  prime  Importance  In  those 
modules  which  Implement  the  major  functions  of  the  system.  With  the  advent 
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We  cbnducted  further  evaluation  of  the  framework  and  the  quality  factors  by 
examining  their  applicability  to  Air  Force  applications.  To  make  this 
evaluation,  a categorization  of  Air  Force  appllcatlohs  was  derived.  This 
categorizdtion  Is  shown  In  Table  3.2>1.  Several  references  [AIRf76,  THEE75, 
SACI76,  SACKH67]  Were  utilized  In  deriving  this  scheme.  We  found,  however, 
that  the  factors  vary  considerably  within  the  categories  as  well  as  between 
categories i depending  on  the  specific  application.  So  we  chose  specific  sys- 
terns  representative  of  the  categories  and  had  several  people  familiar  with 
Air  Force  missions  Identify  the  Importance  of  the  factors  to  the  specific  soft- 
ware product.  The  ratings  shown  In  Table  3.2-2  are  the  means  of  the  Individual 
ratings  and  were  calculated  as  follows: 

V - Very  High  - 6 points 

H - High  - 4 points 

M - Medium  - 3 points 

L - Low  - 1 point 

n 

Average  rating  » ^ '^l 

n 

where  r^  = Individual  ratings 

n « total  nuinber  of  people  rating 
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Table  3.1-1  Definition  of  Software  Quality  Factors 


CORRECTNESS 

Extent  to  which  a program  satisfies  its  specifications 
and  fulfills  the  user's  mission  objectives. 

RELIABILITY 

Extent  to  which  a program  can  be  expected  to  perform 
its  intended  function  with  required  precision. 

EFFICIENCY 

The  amount  of  computing  resources  and  code  required  by 
a program  to  perform  a function. 

INTEGRITY 

Extent  to  which  access  to  software  or  data  by 
unauthorized  persons  can  be  controlled. 

USABILITY 

Effort  required  to  learn,  operate,  prepare  input,  and 
interpret  output  of  a program. 

MAINTAINABILITY 

Effort  required  to  locate  and  fix  an  error  in  an 
operational  program. 

TESTABILITY 

Effort  required  to  test  a program  to  insure  it  performs 

its  intended  function. 

FLEXIBILITY 

Effort  required  to  modify  an  operational  program. 

PORTABILITY 

Effort  required  to  transfer  a program  from  one  hardware 
configuration  and/or  software  system  environment  to 

another. 

REUSABILITY 

Extent  to  which  a program  can  be  used  in  other 
applications  - related  to  the  packaging  and  scope  of  the 
functions  that  programs  perform. 

INTEROPERABILITY 

Effort  required  to  couple  one  system  with  another. 

Table  3.2-1  Categorization  of  Software  in  Air  Force  Systems 
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Rating  In  matrix  was  assigned  according 
to  following: 


u 


The  representative  set  of  missions  Is  shown  against  the  set  of  software  quality 
factors.  The  missions  range  from  a software  test  bed  laboratory  facility  to 
a man-rated  space  mission  (STS).  The  ratings  Illustrate  the  relative  qualities 
that  are  evidenced  by  the  specified  goals  of  the  systems. 

As  shown,  the  factor  of  reliability  Is  absolutely  critical  for  the  success  and 
safety  required  for  the  spacecraft  and  early  warning  applications.  On  the 
other  hand,  while  reliability  Is  always  a desirable  quality,  the  degree  necessary 
for  success  In  the  management  Information  system  and  test  bed  applications 
decreases  relative  to  the  other  missions.  The  amount  of  efficiency  required 
Is  determined  not  only  by  the  application  Itself  but  also  by  the  computing 
resources  available  and  loading  expected  during  a given  mission.  On-board 
processors,  because  of  size  limitations,  usually  have  to  realize  a high  degree 
of  efficiency.  A communication  network,  handling  large  volumes  of  data  at  a 
high  rate,  has  to  be  efficient  In  Its  data  switching  and  handling  capabilities. 

As  another  example,  a large  space  mission  application  dedicated  to  a particular 
hardware  as  well  as  software  environment  may  not  consider  portability  an  essen- 
tial factor,  whereas.  In  the  test  bed  facility,  the  ability  to  transport  soft- 
ware to  and  from  other  configurations  could  well  be  one  of  the  most  Important 
considerations  In  the  entire  system. 
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i In  filling  out  Table  3.2>2.  Me  found  that  very  few  highs  (H)  or  very  highs  (V) 

were  given  to  quality  factors  In  the  product  revision  or  product  transition 
categories.  This  Illustrates  the  lack  of  attention  given  these  areas  In  past 
software  development.  The  average  ratings  were  calculated  to  point  out  this 
fact.  The  quality  factors  associated  with  product  operation  all  received 
high  or  very  high  ratings.  These  were  expected  and  are  probably  justified 
by  the  current  practices  and  procedures  In  use  today  and  utilized  In  specify- 
ing software  development  projects.  The  quality  factors  associated  with 
product  revision  rated  somewhat  lot^r  and.  except  for  a few  high-to-very  high 
and  very  high  ratings  in  maintainability  and  testability,  would  have  been 
even  lower.  The  quality  factors  associated  with  product  transition  all  rated 
medium  to  low.  This,  too,  points  to  the  lack  of  attention  given  to  the 
problems  associated  with  software  conversions. 

There  may  be  more  basic  system  characteristics  that  have  more  effect  on  the 
factors  than  the  categorization  scheme.  A partial  11st  with  the  factors 
that  are  extremely  Important  is  provided  below: 

e If  human  lives  are  affected  (Reliability,  Correctness,  Testability) 

’ f very  high  system  development  cost  (Reliability,  Flexibility) 

• long  life  cycle  (Maintainability,  Portability,  Flexibility) 

a real  time  application  (Efficiency) 

• on-board  application  (Efficiency,  Reliability) 

• processes  classified  Information  (Integrity) 

i • Interrelated  systems  (Interoperability). 

I 3.3  RELATIONSHIP  OF  FACTORS  TO  LIFE-CYCLE  PHASES 

Further  evaluation  of  the  framework  and  quality  factors  was  needed  to  Insure 
^ coverage  of  the  entire  life  cycle  of  a software  product  and  to  determine  If 

I early  Indication  of  the  quality  was  possible.  Table  3.3-1  Identifies  where 

the  factors  should  be  measured  (A  ) and  where  the  greatest  Impact  due  to  poor 
j quality  can  be  expected  (X). 

i 

i For  Instance,  the  reliability  of  a system  Is  Immediately  In  jeopardy  If  a 

contractor  does  not  understand  the  mission  requirements  or  does  not  detect  this 
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fact  during  requirements  analysis  or  system  design  phases.  The  system  may 
' 1 then  proceed  through  code  and  debug  and  even  Into  test  and  operation  before 

the  Impact  of  this  failure  Is  recognized.  The  cost  and  effort  Involved  In 
reevaluation,  redesign,  recoding,  and  retesting  Is  significantly  greater  than 
If  realized  earlier.  Maintainable  and  efficient  code  can  be  achieved  only 
by  designing  and  coding  these  properties  Into  the  software. 

From  the  table  we  see  that  all  of  the  factors  can  be  measured  during  the 
design  phase.  Most  of  the  research  In  metrics  to  date  has  been  In  the  code 
and  debug  area.  Determination  of  quality  Is  more  subjective  In  nature  at 
the  design  level  than  during  the  code  and  debug  phase,  where  metrics  can  be 
applied  to  code.  This  Identifies  where  emphasis  for  research  should  be 
placed  to  take  advantage  of  the  cost  savings  of  early  detection  of  poor 
quality.  As  an  example  of  a more  objective  measure  In  the  design  phase, 
a traceability  matrix,  which  relates  performance  and  design  requirements  to 
the  original  system  requirements  specification,  can  be  effectively  utilized 
to  evaluate  the  quality  of  work  performed  by  the  contractor  during  require- 
ments analysis. 
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DEFINITION  OF  CRITERIA 
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4.1  DEFINING  FACTORS  WITH  CRITERIA 

The  establishment  of  criteria  for  each  factor  has  a four  fold  purpose.  First, 
the  set  of  criteria  for  each  factor  further  defines  the  factor.  Second, 
criteria  which  affect  more  thar.  one  factor  help  describe  the  relationships 
between  factors.  Third,  the  criteria  allow  a one-to-one  relationship  to  be 
established  between  metrics  and  criteria.  Lastly,  the  criteria  further 
establish  the  working  hierarchical  nature  of  the  framework  for  factors  in 
software  quality. 

As  the  software  development  technology  progresses,  other  metrics, 
criteria  or  even  factors  may  be  identified  as  relevant  to  the  needs  of  an 
SPO.  The  framework  being  established  allows  for  and  facilitates  this  kind 
of  expansion  or  refinement  by  its  hierarchical  nature. 


The  set  of  criteria  for  each  quality  factor  are  shown  in  Figure  4.1-1.  The 
factors  are  identified  in  ellipses  and  the  criteria  are  identified  in  rectan- 
gles. These  criteria  were  derived  utilizing  the  software -re la ted  terms  from 
Table  2.4-1,  examining  the  definition  of  each  factor  and  expanding  it  into 
independent  attributes,  and  by  identifying  criteria  with  which  we  can  poten- 
tially associate  objective  measures. 

For  example,  the  attributes  or  standards  for  reliability  are  error  tolerance, 
consistency,  accuracy,  and  simplicity.  Integrity  connotes  protection  which 
implies  two  forms  of  protection:  access  control  and  access  audit. 

The  definitions  of  the  criteria  are  provided  in  Table  4.1-1. 
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Factor 

1 I Criteria 

Figure  4.1-1  Relationship  of  Criteria  to  Software  Quality  Factors  (continued) 
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Table  4.1-1  Criteria  Definitions  for  Software  Quality  Factors 


CRITERION 

DEFINITION 

RELATED 

FACTORS 

TRACEABILITY 

Those  attributes  of  the  software  that  provide 
a thread  from  the  requirements  to  the  imple- 
mentation with  respect  to  the  specific 
development  and  operational  environment. 

Correctness 

COMPLETENESS 

Those  attributes  of  the  software  that 
provide  full  implementation  of  the  functions 
requi red . 

Correctness 

CONSISTENCY 

Those  attributes  of  the  software  that 
provide  uniform  design  and  implementation 
techniques  and  notation. 

Correctness 

Reliability 

Maintainability 

ACCURACY 

Those  attributes  of  the  software  that 
provide  the  required  precision  in  calcula- 
tions and  outputs. 

Reliability 

ERROR  TOLERANCE 

Those  attributes  of  the  software  that 
provide  continuity  of  operation  under 
nonnominal  conditions. 

Reliability 

SIMPLICITY 

Those  attributes  of  the  software  that 
provide  implementation  of-functions  in  the 
most  understandable  manner.  (Usually 
avoidance  of  practices  which  increase 
complexity.) 

Reliability 

Maintainability 

Testability 

MODULARITY 

Those  attributes  of  the  software  that 
provide  a structure  of  highly  independent 
modules. 

Maintainability 

Flexibility 

Testability 

Portability 

Reusability 

Interoperability 

GENERALITY 

Those  attributes  of  the  software  that 
provide  breadth  to  the  functions  performed. 

Flexibility 

Reusability 

EXPANDABILITY 

Those  attributes  of  the  software  that 
provide  for  expansion  of  data  storage 
requirements  or  computational  functions. 

! 

Flexibility 

INSTRUMENTATION 

Those  attributes  of  the  software  that 
provide  for  the  measurement  of  usage  or 
identification  of  errors. 

Testability 

SELF- 

DESCRIPTIVENESS 

Those  attributes  of  the  software  that 
provide  explanation  of  the  implementation 
of  a function. 

Flexibility 

Maintainability 

Testability 

Portability 

Reusability 

Table  4.1-1  Criteria  Definitions  for  Software  Quality  Factors  (Continued) 


CRITERION 

DEFINITION 

RELATED 

FACTORS 

EXECUTION 

EFFICIENCY 

Those  attributes  of  the  software  that 
provide  for  minimum  processing  time. 

Efficiency 

STORAGE 

EFFICIENCY 

Those  attributes  of  the  software  that 
provide  for  minimum  storage  requirements 
during  operation. 

Efficiency 

ACCESS  CONTROL 

Those  attributes  of  the  software  that 
provide  for  control  of  the  access  of 
software  and  data. 

Integrity 

ACCESS  AUDIT 

Those  attributes  of  the  software  that 
provide  for  an  audit  of  the  access  of 
software  and  data. 

Integrity 

OPERABILITY 

Those  attributes  of  the  software  that 
determine  operation  and  procedures  con- 
cerned with  the  operation  of  the  software. 

Usability 

TRAINING 

Those  attributes  of  the  software  that 
provide  transition  from  current  operation 
or  initial  familiarization. 

Usability 

COMMUNICATIVENESS 

Those  attributes  of  the  software  that 
provide  useful  inputs  and  outputs  which 
can  be  assimilated. 

Usability 

SORWARE  SYSTEM 
INDEPENDENCE 

Those  attributes  of  the  software  that 
determine  its  dependency  on  the  software 
environment  (operating  systems,  utilities, 
input/output  routines,  etc.) 

Portability 

Reusability 

MACHINE 

INDEPENDENCE 

Those  attributes  of  the  software  that 
determine  its  dependency  on  the  hardware 
system. 

Portability 

Reusability 

COMMUNICATIONS 

COMMONALITY 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  protocols 
and  interface  routines. 

Interoperability 

DATA  COMMONALITY 

1 

Those  attributes  of  the  software  that 
provide  the  use  of  standard  data  repre- 
sentations. 

Interoperability 

CONCISENESS 

Those  attributes  of  the  software  that 
provide  for  implementation  of  a function 
with  a minimum  amount  of  code. 

Maintainability 

I 
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4.2  RELATIONSHIP  BETWEEN  FACTORS 

The  conceptualization  of  the  factors  In  software  quality  Implies  some  relation- 
ships  between  the  factors.  Those  grouped  under  Product  Operation,  Product 
Revision,  and  Product  Transition  are  related  simply  by  association  with  those 
aspects  of  a software  product's  life  cycle.  These  relationships  are  very  high 
level,  user-oriented  Interactions. 

The  criteria,  especially  those  comnon  to  more  than  one  factor,  provide  a much 
more  detailed  understanding  of  the  factors  and  their  relationships  to  one 
another.  Table  4.2-1  depicts  when  the  criteria  should  be  measured  (a)  and  when 
Impact  of  poor  quality  will  be  realized  (X)  during  the  life-cycle  phases. 

This  table  Is  an  expansion  of  Figure  3.3-1,  the  Impact  of  not  specifying  or 
measuring  software  quality  factors. 

The  effect  of  each  criteria  on  each  factor  was  evaluated  and  the  results  are 
displayed  In  Table  4.2-2.  If  the  existence  of  the  attributes  characterized 
by  the  criterion  positively  Impact  a given  factor,  a (o)  was  placed  In  the  matrix. 
If  the  existence  of  the  attributes  characterized  by  the  criterion  negatively 

. 

Impacts  the  factor,  a (e)  was  placed  In  the  matrix.  If  there  was  no  relationship 
between  the  criterion  and  factor.  If  the  relationship  was  not  clear,  or  If  it 
was  highly  dependent  on  the  application,  a blank  was  placed  In  the  matrix. 

These  criterion/ factor  relationships  are  the  basis  for  the  factor- to-factor 
relationships.  If  all  of  the  criteria  of  one  factor  have  positive  Impacts  on 
, another  factor,  then  the  relationship  of  those  two  factors  Is  very  positive. 

' Conversely,  If  all  of  the  criteria  of  one  factor  have  negative  Impact  on  another 

factor,  these  two  factors  display  a negative  (tradeoff)  relationship. 

j As  an  example,  consider  the  factors  of  portability  and  efficiency.  Two  criteria 

of  efficiency,  execution  efficiency  and  storage  efficiency,  have  negative 
( Impacts  on  portability.  Conversely,  two  criteria  of  portability,  software 
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Table  4.2-1  Impact  of  not  Applying  Criteria  in  Specifying  Software  Quality 


LEGEND:  A Where  criteria  should  be  measured  X Where  impact  of  poor  quality  is  realized 


Tabic  4.2-2  Effect, of  Criteria  on  Software  Quality  Factors 


N.  OMLITY 

N.  FACTORS 

lA 

N. 

Ul 

CRITERIA 

1 

TRACEABILITY 

0 

CONSISTENCY 


ACCURACY 


ERROR  TOLERANCE 


SIMPLICITY 


NODULARITY 


GENERALITY 


EXPANDABILITY 


INSTRUMENTATION 


DESCRIPTIVENESS 


EFFICIENCY 


STORAGE  EFFICIENCY 


ACCESS  CONTROL 


OPERABILITY 


TRAINING 


COMMUNICATIVENESS 


INDEPENDENCE 


MACHINE 


COMMONALITY 


DATA  COMMONALITY 


CONCISENESS 


ACCESS  AUDIT 
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LEGEND  - Attributes  associated  with  criteria  have: 

^Negative  effect  on  quality  factor  QPos^tlve  effect  on  quality  factor 
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system  Independence  and  machine  Independence,  negatively  Impact  efficiency. 

The  relationship  between  efficiency  and  portability  Is  quite  obvious  in  that 
when  a high  degree  of  portability  exists,  you  can  expect  a low  degree  of 
efficiency.  Where  some  criteria  of  a factor  Impact  another  factor  positively 
and  some  negatively,  further  analysis  Is  required  to  determine  the  net  effect 
of  the  Impact. 

Table  4.2-3  displays  the  results  of  the  criteria  versus  factors  analysis  at 
the  factor  versus  factor  level.  In  this  table,  the  (o)  Indicates  that  with 
the  existence  of  a high  degree  of  one  factor  you  would  expect  a high  degree  of 
another  factor.  A (e)  Indicates  that  with  a high  degree  of  one  factor  you 
would  expect  a low  degree  of  the  other  factor. 

Looking  at  portability  versus  efficiency  situation,  we  find  a (e)  entered  In 
the  matrix  shown  in  Table  4.2-3.  This  Indicates  the  tradeoff  situation  already 
discussed.  This  table  then  describes  the  general  relationships  between  factors. 
Specific  cases  must  be  analyzed  along  these  lines  to  determine  the  specific 
tradeoffs.  A discussion  of  the  tradeoffs  generally  found  follows: 

Integrity  vs  Efficiency  - the  additional  code  and  processing  required  to 
control  the  access  of  the  software  or  data  usually  lengthens  run  time 
and  require  additional  storage. 

Usability  vs  Efficiency  - the  additional  code  and  processing  required  to 
ease  an  operator's  task  or  provide  more  usable  output  usually  lengthen 
run  time  and  require  additional  storage. 

Maintainability  vs  Efficiency  - optimized  code.  Incorporating  Intricate 
coding  techniques  and  direct  code,  always  provides  problems  to  the 
maintalner.  Using  modularity.  Instrumentation,  and  well  comnented  high 
level  code  to  Increase  the  maintainability  of  a system  usually  Increases 
the  overhead  resulting  In  less  efficient  operation. 

Testability  vs  Efficiency  - the  above  discussion  applies  to  testing. 
Portability  vs  Efficiency  - the  use  of  direct  code  or  optimized  system 
software  or  utilities  decreases  the  portability  of  the  system,  v 
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Flexibility  vs  Efficiency  - the  generality  required  for  a flexible 
system  Increases  overhead  and  decreases  the  efficiency  of  the  system. 

Reusability  vs  Efficiency  - the  above  discussion  applies  to  reusability. 
Interoperability  vs  Efficiency  - again  the  added  overhead  for  conversion 
from  standard  protocol  and  standard  data  representations,  and  the  use 
of  Interface  routines  decreases  the  operating -efficiency  of  the  system. 

■ 

Flexibility  vs  Integrity  - flexibility  requires  very  general  and  flexible 
data  structures.  This  Increases  the  data  security  problem. 

Reusability  vs  Integrity  - as  In  the  above  discussion,  the  generality 
required  by  reusable  software  provides  severe  protection  problems. 

Interoperability  vs  Integrity  - coupled  systems  allow  for  more  avenues  of 
access  and  different  users  who  can  access  the  system.  The  potential  for 
accidental  access  of  sensitive  data  Is  Increased  as  well  as  the  opportu- 
nities for  deliberate  access.  Often,  coupled  systems  share  data  or  software 
software  which  compounds  the  security  problems  as  well. 
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SECTION  5 


EXAMINATION  OF  SOFTWARE  PRODUCTS  THROUGHOUT  THE  LIFE  CYCLE  PHASES 
5.1  SOFTWARE  PRODUCTS  AS  SOURCES  FOR  METRICS 

One  of  our  main  considerations  In  establishing  metrics  for  the  criteria 
defined  In  the  previous  section  Is  the  availability  of  data  or  software 
products  which  provide  the  sources  for  the  collection  of  the  metrics. 

Software  products  Include  the  source  code  (the  most  obvious  and  researched 
source  of  metrics  to  date),  documentation  Including  requirements  specifi- 
cations. design  specifications,  manuals,  test  plans,  problem  reports  and 
correction  reports,  and  reviews. 

The  most  accurate  source  of  measures  of  the  qualities  of  a software  product 
is  naturally  Its  operational  history.  If  after  two  years  of  operation  a 
software  product  must  be  converted  to  a new  hardware  system  and  the  cost 
to  accomplish  this  Is  100  % of  the  Initial  development  cost,  It  can  be 
stated  that  from  a portability  aspect,  this  product  was  of  poor  quality. 
Testing  provides  very  quantitative  measures  of  the  qualities  of  a software 
product.  The  completeness  of  the  testing  has  always  been  a concern  though. 
Most  testing  Is  oriented  toward  Insuring  the  software  product  runs  as 
efficiently  as  necessary,  performs  functionally  well  (correctness),  and 
does  not  fail  (reliability).  Extensive  testing  usually  also  reveals  the 
usability  and  testability  of  the  product  although  not  often  with  the  people 
who  will  be  using  the  system  or  testing  changes  to  It.  Under  special  cir- 
cumstances tests  may  be  oriented  toward  evaluating  the  Integrity  of  a product. 
In  most  cases  though,  with  the  pressures  of  tight  budgets  and  schedules, 
testing  is  never  as  thorough  as  desired.  Even  when  it  reveals  problems, 
the  costs  to  correct  those  problems  (often  Involving  redesign  as  well  as 
recoding  and  retesting)  are  very  high. 


The  importance  of  finding  errors  earlier  In  the  life  cycle  is  well  known, 
since  the  cost  to  fix  an  error  Increases  rapidly  as  the  life  cycle  prog- 
icsses.  This  is  shown  In  Figure  5.1-1.  In  the  same  manner,  detection  of 
unacceptable  quality  early  is  critical  to  achieving  a high  quality  end 
product. 


It  Is  the  Intent  of  this  study  to  concentrate  on  the  early  phases  of  the 
development  life  cycle  for  metrics  which  will  provide  an  Indication  of 
the  progression  toward  the  desired  product  quality.  The  earlier  In  the 
life  cycle  the  more  "Indicative  only"  these  metrics  will  be.  Obviously 
If  a design  specification  Is  perfectly  written  but  poorly  Implemented  the 
resulting  software  product  will  not  exhibit  a high  level  of  quality.  Metrics 
collected  during  the  implementation  phase  will  help  prevent  or  detect  a 
poor  Implementation.  This  concept  of  successive  application  of  metrics 
Is  explained  further  in  Section  6.  These  concepts  are  Illustrated  In 
Figure  5.1-2. 
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5.2  RANGE  OF  SOFTWARE  PRODUCTS 

While  the  source  code  Is  the  primary  source  for  metrics  during  the  develop 
ment  phase,  documentation  and  reviews  are  generally  available  earlier  and 
If  available  provide  sources  for  metrics.  The  results  of  simulations  at 
the  requirements  analysis  and  design  stages  are  extremely  valuable  Indi- 
cators of  the  quality  of  the  final  product.  However,  because  simulation 
Is  more  of  a tool  to  aid  In  the  development  and  testing  (functional  vali- 
dation) of  the  requirements  and  design  and  is  not  universally  used,  1t 
is  considered  to  be  out  of  the  scope  of  this  study  and  not  a source  for 
formal  metrics.  j 

The  range  of  documents  varies  widely  between  SPO's  and  applications. 
Several  standard  documents  are  required  by  DOD/AP' regulations.  The 
following  references  were  used  to  compile  the  r^-nge  of  documents 
identified  in  Table  5.2-1. 


B0EHB73  Boehm,  B.,  et  al,  "Characteristics  of  Software  Quality", 

Document  #25201 -6001 -RU-00,  NBS  Contract  #3-36012, 

28  December  73. 

C0MP69  "Computer  Program  Development  and  Configuration 

Management",  AFSCF  Exhibit  375-2,  March  69. 


C0MP66a  "Computer  Program  Development  and  Configuration 

Management  for  the  Manned  Orbit  Laboratory  Program", 
SAFSL  Exhibit  20012,  September  66. 

C0MP66b  "Computer  Program  Subsystem  Development  Milestones", 

AFSCF  SSD  Exhibit  o1-47B,  April  66. 

C0NF64  "Configuration  Management  during  Definition  and 

Acquisition  Phases",  AFSCM  375-1,  June  64. 

C0NF66  "Configuration  Management  of  Computer  Prpgrams", 

ESD  Exhibit  EST-1,  Section  H,  April  66.  v'  , 

D0CU74  "Documentation  Standards",  Structured  Programming 

Series  Vol.  VII  and  Addendum,  RADC-TR-74-300,  September 
74  and  April  75. 

D0DM72  "DOD  Directive  for  Automation,  Policy,  Technology, 

and  Standard",  DOD  Manual  4120. 17-M,  December  72. 

HAGAS75  Hagan,  S.,  "An  Air  Force  Guide  for  Monitoring 

and  Reporting  Software  Development  Status," 

NT IS  AD-A016488,  September  75. 
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MILI70  “Military  Standard  Configuration  Management  Practices 

for  Systems*  Equipment,  Munitions  end  Computer  Programs", 
MIL-STD-483,  December  70. 

MILI68  "Military  Standard  Specification  Practices",  MlL-STD- 

490,  October  68. 

PILIM68  Plllglan,  M.S.,  et  a1 , "Configuration  Management  of 

Computer  Program  Contract  End  Items",  ESD-TR-68-107, 
January  68. 

SCH0W76  Schoeffel,  W. , "An  Air  Force  Guide  to  Software  Docu- 

mentation Requirements,"  NTIS  A0-A027  051,  June  76. 

TACT74  "Tactical  Digital  Systems  Documentation  Standards," 

Department  of  the  Navy,  SECNAVINST  3560.1,  August  74. 

The  table  Illustrates  the  considerable  difference  In  the  quantity  of  docu- 
ments among  different  software  system  developments.  The  documents  are 
specified  by  the  AF  regulations  or  SPO-local  regulations  listed  above. 

Each  of  the  document  types  for  a long  llfe/high  cost  software  system  are 
chariicterized  briefly  In  Appendix  8.  Figure  8-1  Identifies  the  docianents 
on  a software  development  timeline  that  were  developed  for  the  two  soft- 
ware systems  which  were  utilized  to  validate  the  metrics.  This  figure 
Identifies  the  most  comprehensive  set  of  documents  required  during  soft- 
ware developments.  Table  B-1  Identifies  which  documents  are  required  by 
or  described  In  each  of  the  references. 

In  order  to  make  the  metrics  widely  applicable,  a representative  set  of 
documents  (sources  for  metrics)  for  any  software  development  had  to  be 
chosen.  It  was  decided  that  there  are  basic  documentation  requirements 
regardless  of  the  size  or  cost  of  the  project.  Certainly  low  cost/short 
life  projects  Incorporate  several  of  the  above  document  characteristics 
Into  one  document.  The  documents  may  not  be  In  as  much  detail.  They  are, 
nevertheless,  desirable  because  of  their  contribution  to  a high  quality 
product.  Thus  while  the  quantity  of  documents  may  vary  with  the  size  of  the 
system  development  effort,  the  documentation  should  provide  the  Ihformatlon 
contained  In  the  documentation  summarized  In  Appendix  B.  If  It  does  not. 
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less  Information  Is  known  about  the  system,  how  It  Is  progressing,  and  later 
how  others  will  maintain,  change,  move  It,  etc.  The  metrics  described  In  the 
next  section  are  not  based  on  the  availability  of  specific  documents  but 
rather  on  the  availability  of  the  Information  contained  In  the  documents. 
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SECTION  6 

DEFINITIONS  OF  METRICS 
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6.1  DEVELOPMENT  OF  METRICS 

The  criteria  were  defined  as  the  attributes  of  the  software  which  provide  or 
determine  a characteristic  of  the  final  software  product.  Metrics  are  defined 
to  provide  a measure  of  these  attributes.  Where  more  than  one  attribute  or 
source  for  metrics  are  found  for  any  one  criterion,  subcriteria  are  established. 

The  subcriteria  further  clarify  the  metric  and  maintain  a one-to-one  relation- 
ship with  the  metrics.  This  further  enhances  the  hierarchial  nature  of  the 
framework  for  factors  in  software  quality. 

Essentially,  there  are  two  types  of  metrics  and  both  are  utilized  in  this  study. 

The  first  type,  like  a ruler,  is  a relative  quantity  measure.  The  second  type 
is  a binary  measure  which  determines  the  existence  (1)  or  absence  (0)  of  some- 
thing. The  units  of  a metric  are  important  to  avoid  ambiguity  and  obtain  a 
meaningful  metric.  The  following  rule  was  used  in  choosing  the  units  of  a metric. 

THE  UNITS  OF  THE  METRIC  WILL  BE 
CHOSEN  AS  THE  RATIO  OF  ACTUAL 
OCCURRENCES  TO  THE  POSSIBLE 
NUMBER  OF  OCCURRENCES. 

Once  stated,  this  rule  seems  obvious,  yet  many  studies  have  failed  because  they 
did  not  comply  with  this  rule.  To  not  comply  results  in  poor  correlation 
between  the  criterion  and  the  factor.  To  illustrate  the  types  of  metrics  and  the 
application  of  the  above  rule,  the  following  two  elementary  examples  are  provided. 

(1)  The  subcriterion,  number  of  unconditional  branches,  could  relate  to 
the  criterion,  simplicity.  Figure  6.1-1  identifies  potential  metrics 
for  this  subcriterion.  The  metric,  the  number  of  GOTO  statements/ 
executable  statements  is  the  most  unambiguous  because  GOTO  state- 
ments are  a proper  subset  of  the  set  of  executable  statements. 

Blank  cards,  comment  cards,  and  declarative  statements  are  i 
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Figure  6.1-1  Choosing  a Metric 


not  potential  GOTO  statements  so  that  the  metrics.  GOTO  statements/ 
cards  and  GOTO  statements/statements.  Invite  ambiguities  to  the 
correlation  of  GOTO  statements  with  the  associated  quality  factor. 

GOTO  statements/routlne  and  GOTO  statements/bi ock  do  not  reveal  the 
size  of  the  set  of  possible  occurrences  and,  therefore,  are  ambiguous 
themselves.  This  Is  an  example  of  the  first  type  of  metric  which 
provides  a relative  quantity  as  a measure.  It  also  provides  a nor- 
malization of  the  metric  so  that  any  measurement  1s  between  0 and  1. 

(2)  The  metric,  use  of  structured  code,  is  an  example  of  a binary  metric 
and  will  be  given  a 1 If  present;  I.e.,  the  target  program  utilizes 
structured  coding  techniques  or  a structured  preprocessor  and. 

If  absent,  a 0.  The  stated  rule  still  holds  In  these  cases  as 
the  presence  of  structured  code/possible  occurrence  of  structure 
code  can  be  viewed  as  a 1/1  = 1 case.  The  binary  case  Is  to  be 
used  where  ambiguity  or  subjectivity  may  enter  the  measurement.  In 
the  example  discussed,  measuring  the  degree  to  which  the  code  Is 
structured  would  be  subjective  and  not  provide  any  enhancement  to 
the  relationship  of  structured  code  to  the  associated  quality 
factor.  A binary  metric  can  also  be  used  to  identify  an  attribute 
(criteria)  which  If  missing  even  once  In  a module  or  system  is  very 
detrimental  to  the  resulting  quality  of  the  software  product. 

Thus  both  type  metrics  are  consistent  with  the  rule  for  choosing  units.  In 
addition,  the  metrics  have  been  chosen  to  be  as  objective  in  nature  as  possible. 
There  are  a few  exceptions.  In  these  cases,  we  have  attempted  to  make  the 
subjective  metric  as  quantitative  and  simple  to  apply  as  possible. 

The  metrics  were  also  chosen  to  be  language  independent.  The  above  two 
examples  are  consistent  with  this  rule.  If  a structured  language  which  did 
not  have  a GOTO- like  construct  was  utilized  both  metrics  would  be  1 indicating 
the  advantage  of  using  that  language. 


Our  process  of  developing  metrics  Involved  the  following  steps: 


(1)  Incorporation  of  work  sponsored  by  RAOC  In  the  areas  of  reliability 
([THAYT76].  CSH00M75],  CSULLJ73]).  portability  (CMEALG68],  [MARSS70]) 
and  maintainability  as  well  as  other  significant  efforts  In  the 
areas  of  metrics  ([B0EHB73].  [ELSHJ76],  [K0SAS74].  [RUBER68], 
[GILBT76])  and  software  design  ([Y0URE75].  [MYERG7S],  [MYERG76], 
[VANT074],  [KERNB74]).  Other  references  are  In  the  Reference  Section. 

(2)  Incorporation  of  metrics  (oriented  primarily  to  source  code  and 
configuration  management)  which  our  management  have  been  using 
for  several  years. 

(3)  Application  of  research  In  complexity  measures  [RICHP76]  and  soft- 
ware physics  ([L0VET76a],  [FITZA76])  that  we  have  conducted  In  the 
past  several  years  and  has  been  conducted  by  others  ([DUNSH77], 
[ELSHJ76].  [HALSM72]). 

(4)  Evaluation  of  a11  of  the  software  products  available  during  a soft- 
ware development  (Section  5)  with  a more  global  view  of  software 
quality  provided  by  the  framework  established  In  the  first  two 
phases  of  this  report. 

(5)  Utilization  of  measures  or  data  available  from  state-of-the-art 
software  support  tools  applicable  during  the  requirements  analysis 
and  design  phases  of  development  ([BR00N76]»  [DAVIC76],  [NBS74], 
[PANZD76],  [N0DA75].  [REIFD76],  [CHANP76],  [RICHP74].  [TIECD76]). 

(6)  Application  of  the  described  concepts  of  a metric. 

The  metrics  established  are  explained  In  paragraph  6.2.  Not  all  of  the 
metrics  Identified  were  supported  by  the  validation.  However,  since  the 
validation  was  performed  with  a limited  sample  and  further  evaluation  and 
experience  Is  required,  the  entire  set  of  metrics  established  are  presented. 

(For  further  discussion,  see  Volume  II.) 


6.2  DESCRIPTION  OF  METRICS 


6.2.1  IDENTIFICATION  OF  METRICS 

6.  f^ers  [MYER676]  states  that  the  "single  major  cause  of  software  errors 
is  mistakes  in  translating  information."  TRW  [THAYT76],  through  an  extensive 
analysis  of  error  data,  states  that  most  "errors  were  design  and  requirements 
errors,  as  opposed  to  coding  errors  and  errors  made  during  the  correction  of 
other  errors."  An  internal  survey  we  conducted  substantiated  these  points 
[L0VET76].  The  documents  and  reviews  discussed  in  section  5 represent  the 
translation  steps  from  users  needs  and  desired  qualities  to  system  imple- 
mentation; requirements  analysis,  design,  implementation.  Our  metrics 
are  oriented  toward  these  steps.  Since  the  metrics  are  indicators  of  the 
progression  toward  good  quality,  we  found  that  in  some  cases  a metric 
would  be  applied  at  all  three  phases  of  development.  Each  measurement 
constitutes  a unique  metric.  Thus,  during  the  requirements  phase,  collec- 
tion of  a set  of  metrics  will  provide  a measure  of  the  progression  toward 
the  desired  levels  of  quality  at  that  point  in  time.  The  metrics  estab- 
lished are  identified  in  this  manner  (according  to  phases)  and  in  relation- 
ship to  the  criteria/subcriteria  in  Table  6.2-1. 

The  metrics  can  be  applied  at  two  levels  in  most  cases,  at  the  module 
level  or  the  system  level.  The  SPO  will  utilize  the  metrics  at  the 
system  level  to  obtain  an  overall  measure  of  how  the  system  is  progressing 
with  respect  to  a particular  quality  factor.  At  the  system  level  the  SPO 
will  be  interested  in  which  metrics  or  metric  elements  scores  are  unusually 
low.  It  may  be  there  is  a general  failure  to  comply  with  a standard  by 
all  designers.  Corrective  action  could  be  taken  to  alleviate  this  type  of 
problem.  The  SPO  may  also  be  interested  in  a module  by  module  metric 
value.  A particular  metric's  low  score  at  a system  level  may  not  be  caused 
by  a general  failure  but  rather  by  particular  modules  having  unusually 
low  scores.  In  this  case  the  SPO  would  want  to  be  aware  of  which  modules 
are  problems  so  that  particular  corrective  actions  or  more  emphasis  could 
be  placed  on  the  development  of  those  modules.  The  metrics  which  cannot 
be  applied  at  both  levels  are  noted  in  the  table. 
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To  Illustrate  how  this  table  should  be  read,  a few  examples  will  be  dis- 
cussed. The  first  metric  Identified  In  the  table  corresponds  to  the  cri- 
terion, traceability.  The  metric  Is  the  number  of  Itemized  requirements 
traced  divided  by  the  total  number  of  Itemized  requirements.  This  metric 
has  significance  and  can  be  measured  at  two  points  In  time,  during  the 
design  and  Implementation  phases.  During  the  design  phase,  the  Identification 
of  which  Itemized  requirements  are  being  satisfied  In  the  design  of  a module 
should  be  documented.  A traceability  matrix  Is  one  way  of  providing  this 
trace.  During  the  Implementation  phase.  Identification  of  which  portions 
of  code  Implement  each  Itemized  requirement  should  be  made.  Some  form  of 
automated  notation,  prologue  comments  or  Imbedded  comments.  Is  comnonly  used 
to  provide  the  trace  at  this  level.  The  bold  line  boxes  under  design  and 
Implementation  value  columns  represent  these  two  measurements.  The  value 
columns  are  used  because  both  measurements  are  relative  quantity  or  calculated; 
the  ratio  of  the  Itemized  requirements  traced  to  total  number  of  requirements. 
A metric  which  Is  a binary  measure  would  be  placed  In  a bold  line  box  In 
the  yes/no  column.  An  example  of  a binary  measure  Is  SI. 2,  the  use  of  a 
structured  language  or  a structured  preprocessor  (paragraph  6. 2. 2. 6).  A 
1 or  0 would  be  placed  In  the  yes/no  column  for  this  metric.  Some  metrics 
are  checklist  type  metrics  In  that  there  are  several  elements  which  must  be 
measured  and  then  summarlzed/normallzed.  The  sunmarlzatlon/nonnallzatlon  Is 
done  In  the  summary  line  which  Is  labeled.  Metric  Value.  To  be  consistent, 
the  final  value  of  each  metric  Is  placed  In  the  sunmary  line.  The  next 
example  will  Illustrate  a combination  of  these  different  situations. 


The  metric  associated  with  the  design  structure  relative  to  the  criterion 
simplicity  (see  page  6-12)  Illustrates  the  various  combinations  a metric 
can  assume.  Several  elements  make  up  the  metric.  These  elements  are 
measured  In  the  design  and  Implementation  phases.  It  Is  possible  to  have 
a metric  which  Is  only  measured  during  one  phase.  Some  of  the  elements  are 
applied  by  a binary  measure  (yes/no  column)  and  some  are  applied  by  a relative 
quantity  measure.  In  which  case  the  measure  Is  Indicated  In  parentheses. 

The  overall  metric  value  for  the  system  Is  Indicated  In  the  sunnary  line. 
Explanations  of  each  of  the  metrics  and  metric  elements  can  be  found  In  the 
following  paragraphs.  Further  Information  and  examples  are  available  In 
Section  8 and  Appendix  D,  where  collecting  metric  data  Is  described. 
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Table  6.2-1  Software  Quality  Metrics 
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Table  6.2-1  Software  Quality  Metrics  (Continued) 
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Table  6.2-1  Software  Quality  Metrics  (Continued) 


IB 


SYSTEM  Sum  of  wodule  scores 
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Table  6.2-1  Software  Quality  Metrics  (Continued) 


MAINTAINABILITY,  FLEXIBILITY, 
TESTABILITY,  PORTABILITY, 
FACTOR(S):  REUSABILITY,  INTEROPERABILITY 


METRIC  VALUE  A {cable  elements 
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Table  6.2-1  Software  Quality  Metrics  (Continued) 


ii. 


(0  iA 

O U 

o 

* 1.  4) 

D.  »- 
Q. 

2 4)  «0 

a c 
s.  c o 
4)  i/» 

•M  <0 
C C 
O t- 
U 

O (A 

*0  C 

. C ® 

Sb  ^ 

O 

V • u 

« 

4)  *r-  V. 
Q.TI  O 
O Q 

U ® 2 

0*4) 
V-  4)  O. 

> O 

«/)  <d 

c U)  4- 
0^0 

M (/>  U 

a 41 

> 4-)  jQ 

O <o  S 
> ^ a 

Ol  tn  Z 


U • 4) 

U 9 V) 

^ c 

4)  C O 


a « *o 

'DEC 
4)  <0 

O iA 


& ^ E 

O 4)  O 
»-  TJ  »•- 
4)  •*-  C 

> • > T- 

■SC  e o 

4)  Q.  ‘r- 

^ C 4-» 

<0  lA  iA 
)0  4)  O 

U 4-»  lA  C 

4)  C •r-  0> 

u to 

10  m U •f 
E E 4)  T3 
_ K 
O)  * 4)  *0 
CM  e 

•w  L.  TJ  <0 
C « O 
M 4-»  • 

« a io  a. 

• • Sb  ^ ^ • 

H-  *Q  3 4)  r 

00  C E -C 

-J  e M 

Z lO  * 4-> 

U>  ^ Y*  D C 
UJ  & t.  4)  r 

X O 4->  ’r-  j 

U C M U I 

O <0  •r-  r 

CJ)  M i-  p-  4.  1 

X M 4J  10  < 

w-4  4)  CL  4)  3 ; 

Z ^ O X to  ( 

S ^ ^ 


L 


llcable  elements 
elements 


Software  Quality  Metrics  (Continued) 


FACTOR(S);  PORTABILITY.  REUSA8ILITY 


FACTOR(S):  INTEROPERABILITY 


FACTOR(S):  MAIMTAINABILITY 


6.2.2  EXPLANATIONS  OF  METRICS 

Each  metric  and  each  metric  element  are  described  in  the  following  paragraphs. 
Indication  is  provided  if  the  metric  is  applied  at  the  system  level  or  the 
module  level  and  during  which  phases. 
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6. 2. 2.1  Traceability 

TR.l  Cross  reference  relating  modules  to  requirements  (design  and  imple- 
mentation phases  at  system  level). 

During  design,  the  identification  of  which  itemized  requirements  are  satis- 
fied in  the  design  of  a module  are  documented.  A traceability  matrix  is  an 
example  of  how  this  can  be  done.  During  implementation,  which  itemized  require- 
ments are  being  satisfied  by  the  module  implementation  are  to  be  identified. 

Some  form  of  automated  notation,  prologue  conments  or  imbedded  comments,  is 
used  to  provide  this  cross  reference.  The  metric  is  the  number  of  itemized 
requirements  traced  divided  by  the  total  number  of  itemized  requirements. 

6. 2. 2. 2 Completeness 

CP.l  Completeness  Checklist  (All  three  phases  at  system  level). 

This  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 


(1)  Unambiguous  references  (input,  function,  output). 

Unique  references  to  data  or  functions  avoid  ambiguities  such  as 
a function  being  called  one  name  by  one  module  and  by  another 
name  by  another  module.  Unique  references  avoid  this  type  of 
ambiguity  in  all  three  phases. 


(2)  All  data  references  defined,  computed,  or  obtained  from  an 
external  source. 

Each  data  element  is  to  have  a specific  origin.  At  the 
requirements  level  only  major  global  data  elements  and  a few 
specific  local  data  el«nents  may  be  available  to  be  checked. 

The  set  of  data  elements  available  for  completeness  checking  at 
the  design  level  increases  substantially  and  is  to  be  complete 
at  implementation. 
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(3)  All  defined  functions  used. 

A function  which  is  defined  but  not  used  during  a phase  1$ 
either  nonfunctional  or  a reference  to  it  has  been  oinitted. 


(4)  All  referenced  functions  defined. 

A system  Is  not  complete  at  any  phase  if  dummy  functions  are 
present  or  if  functions  have  been  referenced  but  not  defined. 

(5)  All  conditions  and  processing  defined  for  each  decision  point. 

Each  decision  point  is  to  have  all  of  its  conditions  and  alter- 
native processing  paths  defined  at  each  phase  of  the  software 
development.  The  level  of  detail  to  which  the  conditions  and  alter- 
native processing  are  described  may  vary  but  the  important  element 
is  that  all  alternatives  are  described. 

(6)  All  defined  and  referenced  calling  sequence  parameters  agree. 

For  each  interaction  between  modules,  the  full  complement  of 
defined  parameters  for  the  interface  Is  to  be  used.  A par- 
ticular call  to  a module  should  not  pass,  for  example,  only  five 
of  the  six  defined  parameters  for  that  module. 

(7)  All  problem  reports  resolved. 

At  each  phase  In  the  development,  problem  reports  are  generated. 

Each  is  to  be  closed  or  a resolution  indicated  to  ensure  a 
complete  product. 

(8)  Design  agrees  with  requirements. 

Continual  updating  of  the  requirements  documentation  and  the 
design  documentation  is  required  so  that  the  current  version  of 
the  source  code  (see  element  (9)),  the  current  version  of  the 
design  documentation,  and  the  current  version  of  the  require- 
ments documentation  agreee. 


f J 


(9)  Code  agrees  with  design. 

See  element  (8). 

6. 2. 2. 3 Consistency 

CS.l  Procedure  Consistency  Measure  (design  and  implementation  at  system 
level ) . 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Standard  Design  Representation. 

Flow  charts,  HIPO  charts.  Program  Design  Language  - whichever  form 
of  design  representation  is  used,  standards  for  representing  the 
elements  of  control  flow  are  to  be  established  and  followed.  This 
element  applies  to  design  only.  The  measure  is  based  on  the  number  of 
modules  whose  design  representation  does  not  comply  with  the  standards. 

(2)  Calling  sequence  conventions. 

Interactions  between  modules  are  to  be  standardized.  The  stan- 
dards are  to  be  established  during  design  and  followed  during 
implementation.  The  measure  is  based  on  the  number  of  modules 
which  do  not  comply  with  the  conventions. 

(3)  Input/Output  Conventions. 

Conventions  for  which  modules  will  perform  I/O,  how  it  will  be 
accomplished,  and  the  I/O  formats  are  to  be  established  and 
followed.  The  measure  is  based  on  which  modules  do  not  comply  with 
the  conventions. 

(4)  Error  Handling  Conventions. 

A consistent  method  for  error  handling  is  required.  Conven- 
tions established  in  design  are  followed  into  implementation. 

The  measure  is  based  on  the  number  of  modules  which  do  not 
comply  with  the  conventions. 


CS.2  Data  Consistency  Measure  (Design  and  Implementation  at  system  level) 

The  metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Standard  data  usage  representation. 

In  concert  with  CS.1  (1).  a standard  design  representation  for 
data  usage  Is  to  be  established  and  followed.  This  Is  a design  metric 
only.  Identifying  the  number  of  modules  which  violate  the  standards. 

(2)  Naming  Conventions. 

Naming  conventions  for  variables  and  modules  are  to  be  established 
and  followed. 

(3)  Unit  Consistency. 

Units  of  variables  are  to  be  chosen  to  be  consistent  with  a11 
uses  of  the  variables.  The  measure  Is  based  on  the  number  of 
modules  In  which  consistent  units  are  not  utilized.  This  can  be 
measured  at  both  design  and  Implementation. 

(4)  Consistent  Global  Definitions. 

Global  data  elements  are  to  be  defined  In  the  same  manner  by  a11 
modules.  The  measure  Is  based  on  the  number  of  modules  In  which 
the  global  data  elements  are  defined  In  an  Inconsistent  manner 
for  both  design  and  Implementation. 

(5)  Data  Type  Consistency. 

A data  element  defined  as  a particular  data  type  is  to  be  used  as  that 
data  type  in  all  occurrences.  A common  violation  of  this  rule  is  found 
in  arrays  where  several  data  types  are  defined.  The  measure  is  based 
on  the  number  of  modules  which  utilize  data  types  Inconsistently. 


6. 2. 2. 4 Accuracy 

AV.1  Accuracy  Checklist  (requirements,  design.  Implementation  phases  at 
system  level).  Each  element  Is  a binary  measure  Indicating  existence,  or 
absence  of  the  elements.  The  metric  Is  the  sum  of  the  scores  of  the 
following  applicable  elements  divided  by  the  number  of  applicable  elements. 

(1)  Error  analysis  performed  and  budgeted  to  module  (requirements 
phase  only). 

An  error  analysis  must  be  part  of  the  requirements  analysis  performed 
to  develop  the  requirements  specification.  This  analysis  allocates 
overall  accuracy  requirements  to  the  Individual  functions  to  be 
performed  by  the  system.  This  budgeting  of  accuracy  requirements 
provides  definitive  objectives  to  the  module  designers  and 
implementers. 

(2)  A definitive  statement  of  requirement  for  accuracy  of  inputs, 
outputs,  processing,  and  constants  (requirements  phase  only). 

See  explanation  above  (1). 

(3)  Sufficiency  of  Math  Library  (design  phase  only). 

The  accuracy  of  the  math  library  routines  utilized  within  the 
system  is  to  be  checked  for  consistency  with  the  overall 
accuracy  objectives. 


(4)  Sufficiency  of  numerical  methods  (design  and  implementation 
phase ) ■ 

The  numerical  methods  utilized  within  the  system  are  to  be  consis- 
tent with  the  accuracy  objectives.  They  can  be  checked  at  design 
and  implementation. 

(5)  Execution  outputs  within  tolerances  (implementation  phase  only 
requiring  execution). 

A final  measure  during  development  testing  is  execution  of  mod- 
ules and  checking  for  accuracy  of  outputs. 
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6. 2.2. 5 Error  Tolerance 

ET.l  Error  Tolerance  Control  Checklist  (design  and  implementation  phases 
at  system  level). 

The  metric  is  the  sum  of  the  scores  given  to  the  following  elements  divided 
by  the  number  of  applicable  elements. 

(1)  Concurrent  processing  centrally  controlled. 

Functions  which  may  be  used  concurrently  are  to  be  controlled 
centrally  to  provide  concurrency  checking,  read/write  locks,  etc. 
Examples  are  a data  base  manager,  I/O  handling,  error  handling, 
etc.  The  central  control  must  be  considered  at  design  and  then 
implemented. 

(2)  Errors  fixable  and  processing  continued. 

When  an  error  is  detected,  the  capability  to  correct  it  on-line 
and  then  continue  processing,  should  be  available.  An  example  is 
an  operator  message  that  the  wrong  tape  is  mounted  and  processing 
will  continue  when  correct  tape  is  mounted.  This  can  be  measured 
at  design  and  implementation. 

(3)  When  an  error  condition  is  detected,  the  condition  is  to  be  passed  up  to 
calling  routine. 

The  decision  of  what  to  do  about  an  error  is  to  be  made  at  a 
level  where  an  affected  module  is  controlled.  This  concept  is 
built  into  the  design  and  then  implemented. 

ET.2  Recovery  from  Improper  Input  Data  Checklist  (all  three  phases  at 
system  level).  The  metric  is  the  sum  of  the  scores  of  the  following  appli- 
cable elements  divided  by  the  number  of  the  applicable  elements. 


t 

« 


» 


I 


t 


(1)  A definitive  statement  of  requirement  for  error  tolerance  of 
input  data. 

The  requirements  specification  must  identify  the  error  tolerance 
capabilities  desired  (requirements  phase  only). 

(2)  Range  of  values  (reasonableness)  for  items  specified  and  checked 
(design  and  implementation  phases  only). 

The  attributes  of  each  input  item  are  to  be  checked  for  reason- 
ableness. Examples  are  checking  items  if  they  must  be  numeric, 
alphabetic,  positive  or  negative,  of  a certain  length,  nonzero, 
etc.  These  checks  are  to  be  specified  at  design  and  exist  in 
code  at  implementation. 

(3)  Conflicting  requests  and  illegal  combinations  identified  and  checked 
(design  and  implementation  phases  only). 

Checks  to  see  if  redundant  input  data  agrees,  if  combinations  of  param- 
eters are  reasonable,  and  if  requests  are  conflicting  should  be  docu- 
mented in  the  design  and  exist  in  the  code  at  implementation. 

(4)  All  input  is  checked  before  processing  begins  (design  and  imple- 
mentation phases  only). 

Input  checking  is  not  to  stop  at  the  first  error  encountered  but  to  con- 
tinue through  all  the  input  and  then  report  all  errors.  Processing  is 
not  to  start  until  the  errors  are  reported  and  either  corrections  are 
made  or  a continue  processing  command  is  given. 

(5)  Determination  that  all  data  is  available  prior  to  processing. 

To  avoid  going  through  several  processing  steps  before  incomplete 
input  data  is  discovered,  checks  for  sufficiency  of  input  data 
are  to  be  made  prior  to  the  start  of  processing. 


ET.3  Recovery  from  Computational  Failures  Checklist  (all  three  phases  at 
system  level).  The  metric  is  the  sum  of  the  scores  of  the  following  appli- 
cable elements  divided  by  the  number  of  applicable  elements. 
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(1)  A definitive  statement  of  requirement  for  recovery  from  compu- 
tational failures  (requirements  phase  only). 

The  rtqulrfemant  for  this  type  error  tolerance  capability  are  to 
be  stated  during  requirements  phase. 

(2)  Loop  and  multiple  transfer  Index  parameters  range  tested  before 
use.  (design  ahd  implementation  phase  only). 

Range  tests  for  loop  Indices  and  multiple  transfers  are  to  be 
specified  at  design  and  to  exist  In  code  at  Implementation. 

(3)  Subscript  checking  (design  and  Implementation  phases  only). 

Checks  for  legal  subscript  values  are  to  be  specified  at  design 
and  coded  during  Implementation. 

(4)  Critical  output  parameters  reasonableness  checked  during 
processing  (design  and  Implementation  phases  only). 

Certain  range-of- value  checks  are  to  be  made  during  processing  to 
ensure  the  reasonableness  of  final  outputs.  This  Is  usually  done 
only  for  critical  parameters.  These  are  to  be  Identified  during 
design  and  coded  during  Implementation. 

ET.4  Recovery  from  Hardware  Faults  Checklist  (All  three  phases  at  system 
level).  The  metric  is  the  sum  of  scores  from  the  applicable  elements  divided 
by  the  number  of  applicable  elements. 

(1)  A definitive  statement  of  requirements  for  recovery  from  hardware 
faults  (requirements  only). 

The  handling  of  hardware  faults  such  as  arithmetic  faults,  power 
failure,  clock  interrupts,  etc.,  are  to  be  specified  during  require- 
ments phase. 
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(2)  Recovery  from  Hardware  Faults  (design  and  Implementation  phases 
only). 

The  design  specification  and  code  to  provide  the  recovery  from 
the  hardware  faults  Identified  In  the  requirements  must  exist 
In  the  design  and  Implementation  phases  respectively. 

ET.5  Recovery  from  Device  Errors  Checklist  (all  three  phases  at  system 
level).  The  metric  Is  the  score  given  to  the  applicable  elements  below 
at  each  phase. 

(1)  A definitive  statement  of  requirements  for  recovery  from  device 
errors  (requirements  only). 

The  handling  of  device  errors  such  as  unexpected  end-of-flles 
or  end-of-tape  conditions  or  read/write  failures  are  specified 
during  the  requirements  phase. 

(2)  Recovery  from  Device  Errors  (design  and  implementation  phases 
only). 

The  design  specification  and  code  to  provide  the  required 
handling  of  device  errors  must  exist  in  the  design  and  implementation 
phases  respectively. 

6. 2. 2. 6 Simplicity 

SI.l  Design  Structure  Measure  (design  and  implementation  phases  at  system 
level).  The  metric  is  the  sum  of  the  scores  of  the  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Design  organized  in  top  down  fashion. 

A hierarchy  chart  of  system  modules  Is  usually  available  or  easy 
to  construct  from  design  documentation.  It  should  reflect  the 
accepted  notion  of  top  down  design.  The  system  is  organized 
in  a hleracrchal  tree  structure,  each  level  of  the  tree  represents 
lower  levels  of  detail  descriptions  of  the  processing. 
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(2)  No  duplicate  functions- 

Descriptions  of  functions  to  be  performed  by  each  module  at 
design  and  the  actual  function  performed  by  the  coded  module 
Is  to  be  evaluated  to  ensure  It  Is  not  duplicated  by  other 
modules. 

(3)  Module  Independence. 

The  processing  done  within  a module  Is  not  to  be  dependent  on  the 
source  of  Input  or  the  destination  of  the  output.  This  rule  can 
be  applied  to  the  module  description  during  design  and  the  coded 
module  during  Implementation.  The  measure  for  this  element  Is 
based  on  the  number  of  modules  which  do  not  comply  with  this  rule. 

(4)  Module  processing  not  dependent  on  prior  processing. 

The  processing  done  within  a module  is  not  to  be  dependent  upon 
knowledge  or  results  of  prior  processing,  e.g.,  the  first  time 
through  the  module,  the  nth  time  through,  etc.  This  rule  is 
applied  as  above  at  design  and  implementation. 

(5)  Each  module  description  Includes  Input,  output,  processing, 
limitations. 

Documentation  which  describes  the  Input,  output,  processing,  and 
limitations  for  each  module  is  to  be  developed  during  design  and 
available  during  Implementation.  The  measure  for  this  element  is 
based  on  the  number  of  modules  which  do  not  have  this  Information 
documented. 

(6)  Each  module  has  single  entrance,  single  exit. 

Determination  of  the  number  of  modules  that  violate  this  rule  at 
design  and  implementation  can  be  made  and  Is  the  basis  for  the  metric. 

(7)  No  global  data. 

This  Is  a binary  measure  which  Identifies  the  complexity  added  to  a 
system  by  the  use  of  global  data.  If  no  global  data  exists,  this 
measure  Is  1,  If  global  data  does  exist.  It  Is  0. 
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SI. 2 Use  of  structured  language  or  structured  language  preprocessor  (imple- 
mentation phase).  The  metric  is  a binary  measure  of  the  existence  (1) 
or  absense  (0)  of  structured  language  code. 

A structured  language  or  structured  language  preprocessor  provides  con- 
structs similar  to  the  IFTHENELSE,  DOWHILE,  DOUNTIL,  and  CASE  statements 
associated  with  structured  programming. 

SI.  3 Data  and  Control  Flow  Complexity  measure  (Design  and  implementation 
phases) . 

This  metric  can  be  measured  from  the  design  representation  (e.g.,  flowcharts) 
and  the  code  automatically.  Path  flow  analysis  and  variable  set/use  informa- 
tion along  each  path  is  utilized.  A variable  is  considered  to  be  'live'  at  a 
node  if  it  can  be  used  again  along  that  path  in  the  program.  The  com- 
plexity measure  is  based  on  sunming  the  'liveness'  of  all  variables  along 
all  paths  in  the  program.  It  is  normalized  by  dividing  it  by  the  maximum 
complexity  of  the  program  (all  variables  live  along  all  paths). 

(See  [RICHP76]  and  page  D-16  of  Volume  II.) 

SI. 4 Measure  of  Simplicity  of  Coding  Techniques  (Implementation  phase 
applied  at  module  level  first).  The  metric  at  the  system  level  is  an 
averaged  quantity  of  all  the  module  measures  for  the  system.  The  module 
measure  is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Module  flow  top  to  bottom. 

This  is  a binary  measure  of  the  logic  flow  of  a module.  If  it 
flows  top  to  bottom,  it  is  given  a value  of  1,  if  not  a 0. 

(2)  Negative  Boolean  or  complicated  Compound  Boolean  expressions 
used. 

Compound  expressions  involving  two  or  more  Boolean  operators  and 
negation  can  often  be  avoided.  These  types  of  expressions  add 
to  the  complexity  of  the  module.  The  measure  is  based  on  the 
number  of  these  complicated  expressions  per  executable  statement 
in  the  module. 
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(3)  Jumps  In  and  out  of  loops. 

Loops  within  a module  should  have  one  entrance  and  one  exit. 

This  measure  Is  based  on  the  number  of  loops  which  comply  with  this 
rule  divided  by  the  total  number  of  loops. 

(4)  Loop  Index  modified. 

Modification  of  a loop  1nde;(  not  only  complicates  the  logic  of  a 
module  but  causes  severe  problems  while  debugging.  This  measure 
Is  based  on  the  number  of  loop  Indices  which  are  modified  divided 
by  the  total  number  of  loops. 

(5)  Module  Is  not  self-modifying. 

If  a module  has  the  capability  to  modify  Its  processing  logic  It  becomes 
very  difficult  to  recognize  what  state  It  Is  In  when  an  error  occurs.  In 
addition,  static  analysis  of  the  logic  Is  more  difficult.  This  measure 
emphasizes  the  added  complexity  of  self-modifying  modules. 

(6)  All  arguments  passed  to  a module  are  parametric. 

This  Is  a binary  measure,  1 If  all  parameters  are  parametric, 

0 If  they  all  are  not.  This  measure  1$  based  on  the  .Jtentlal 
problems  that  can  arise  If  constants  or  global  data  are  used  as 
arguments. 

(7)  Number  of  statement  labels.  ^ 

This  measure  Is  based  on  the  premise  that  as  more  statement  labels 
are  added  to  a module  the  more  complex  It  becomes  to  understand. 


(8)  Unique  names  for  variables. 

This  Is  a binary  measure  which  Is  given  a 1 If  unique  names  are 
used,  and  a 0 If  they  are  not. 
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(9)  Single  use  of  variables. 

A variable  Is  to  be  utilized  for  only  one  purpose,  I.e.,  in  one 
manner.  This  measure  Is  a binary  measure;  1 If  variables  are 
used  In  only  one  way  and  0 If  they  are  used  for  multiple  purposes. 

(10)  No  mixed-mode  expressions. 

If  mix-mode  expressions  are  used  greater  complexity  Is  Introduced. 
This  measure  Is  a 1 if  no  mix-mode  expressions  are  used  In  a 
module,  and  a 0 If  mix-mode  expressions  are  used. 

(11)  Nesting  level . 

The  greater  the  nesting  level  of  decisions  or  loops  within  a mod- 
ule, the  greater  the  complexity.  The  measure  Is  the  Inverse  of 
the  maximum  nesting  level. 

( 12)  Number  of  branches . 

The  more  paths  or  branches  that  are  present  In  a module,  the 
greater  the  complexity.  This  measure  Is  based  on  the  number  of 
decision  statements  per  executable  statements. 

(13)  Number  of  GOTO' s. 

Much  has  been  written  In  the  literature  about  the  virtues  of 
avoiding  GOTO's.  This  measure  Is  based  on  the  number  of  GOTO 
statements  per  executable  statement. 

(14)  No  extraneous  code  exists. 

This  Is  a binary  measure  which  Is  1 If  no  extraneous  code  exists 
and  0 If  It  does.  Extraneous  code  Is  code  which  Is  nonfunctional 
or  cannot  be  executed. 
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(15)  Variable  mix  In  a module. 

From  a s1mp11c1t;y  viewpoint,  local  variables  are  far  better  than 
global  variables.  This  measure  1s  the  ratio  of  Internal  (local) 
variables  to  total  (Internal  (local;  plus  external  (global)) 
variables  within  a module. 

(16)  Variable  density. 

The  more  uses  of  variables  In  a module  the  greater  the  complexity 
of  that  module.  This  measure  Is  based  on  the  number  of  variable 
uses  In  a module  divided  by  the  maximum  possible  uses. 

6.2.2. 7 Modularity 

N0.1  Stability  Measure  (Design  phase  at  system  level). 

This  measure  Is  based  on  G.  Meyer's  ([MYERG76])  categorization  of  modules  by 
their  module  strength  and  coupling.  Module  strength  Is  a measure  of  the  cohe- 
siveness or  relationship  of  the  elements  within  a module.  Module  coupling  Is 
a measure  of  the  relationship  between  modules.  The  metric  combines  these  two 
measures  to  calculate  the  expected  nimiber  of  modules  that  would  require  modifi- 
cation If  changes  to  any  one  module  were  made  divided  by  the  total  number  of 
modules. 

M0.2  Modular  Implementation  Measure  (design  and  Implementation  phases  at  sys- 
tem level).  The  metric  Is  the  sum  of  the  scores  of  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 

(1)  Hierarchical  Structure. 

The  measure  refers  to  the  modular  Implementation  of  the  top  down  design 
structure  mentioned  In  SI.l  (1).  The  nierarchical  structure  obtained 
should  exemplify  the  following  rules;  Interactions  between  modules 
are  restricted  to  flow  of  control  between  a predecessor  module  and  Its 
Immediate  successor  modules.  This  measure  Is  based  on  the  number  of 
violations  to  this  rule. 

(2)  All  modules  do  not  exceed  a standard  module  size  (100)  (Implementa- 
tion phase  only). 

The  standard  module  size  of  100  procedural  statements  can  vary.  100 
was  chosen  because  It  has  been  mentioned  In  the  literature  frequently. 
This  measure  Is  based  on  the  number  of  modules  which  exceed  the 
standard  size  established. 
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(3)  All  modules  represent  one  function. 

The  concept  of  modularity  Is  based  on  each  function  being  Imple- 
mented In  a unique  module.  This  measure  Is  based  on  the  number 
of  modules  which  represent  more  than  one  function.  This  can  be 
determined  at  both  design  and  Implementation. 

(4)  Controlling  parameters  defined  by  calling  module. 

The  next  four  elements  further  elaborate  on  the  control  and 
Interaction  between  modules  referred  to  by  (1)  above.  The 
calling  module  defines  the  controlling  parameters,  any  Input 
data  required,  and  the  output  data  required.  Control  must  also  be 
returned  to  the  calling  module.  This  measure  and  the  next  three  are 
based  on  the  number  of  violations  to  these  rules.  They  can  be 
measured  at  design  and  Implementation. 

(5)  Input,  data  controlled  by  calling  module. 

See  (4)  above. 

(6)  Output  data  provided  to  calling  module. 

See  (4)  above. 

(7)  Control  returned  to  calling  module. 

See  (4)  above. 


(8)  Modules  do  not  share  temporary  storage. 

i This  is  a binary  measure,  1 if  modules  do  not  share  temporary 

j storage  and  0 If  they  do.  It  emphasizes  the  loss  of  module  Inde- 

^ pendence  if  temporary  storage  Is  shared  between  modules. 

I 6.2. 2. 8 Generality 

* GE.l  Extent  to  which  modules  are  referenced  by  other  modules  (design  and 

Implementation  at  system  level).  This  metric  provides  a measure  of  the  gen- 
erality of  the  modules  as  they  are  used  In  the  current  system.  A module 
< Is  considered  to  be  more  general  in  nature  If  It  Is  used  (referenced)  by 

more  than  one  module.  The  number  of  these  common  modules  divided  by  the 
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liE.  2 Implementation  for  Generality  Measure  (design  and  Implementation 
phases).  This  metric  Is  the  sum  of  the  scores  of  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 

(1)  Input,  processing,  output  functions  are  not  mixed  In  a single 
function. 

A module  which  performs  I/O  as  well  as  processing  Is  not  as 
general  as  a module  which  simply  accomplishes  the  processing. 

This  measure  Is  based  on  the  number  of  modules  that  violate 
this  concept  at  design  and  Implementation. 

(2)  Application  and  machine  dependent  functions  are  not  mixed  In 
a single  module  (Implementation  only). 

Any  references  to  machine  dependent  functions  within  a module 
lessens  Its  generality.  An  example  would  be  referencing  the 
system  clock  for  timing  purposes.  This  measure  Is  based  on  the 
number  of  modules  which  violate  this  concept  at  design  and 
I Implementation. 

(3)  Processing  not  data  volume  limited. 

A module  which  has  been  designed  and  coded  to  accept  no 
more  than  100  data  Item  Inputs  for  processing  Is  certainly 

. not  as  general  In  nature  as  a module  which  will  accept  any 

i volume  of  Input.  This  measure  Is  based  on  the  number  of  modules 

which  are  designed  or  Implemented  to  be  data  volume  limited. 

I 

(4)  Processing  not  data  value  limited. 

A previously  Identified  element,  ET. 2 (2)  of  Error  Tolerance  dealt 
with  checking  Input  for  reasonaoleness.  This  capability  Is  required 
to  prevent  providing  data  to  a function  for  which  It  Is  not  defined 
or  Its  degree  of  precision  Is  not  acceptable,  etc.  This  .s  necessary 
capability  from  an  error  tolerance  viewpoint.  From  a generality 
viewpoint,  the  smaller  the  subset  of  all  possible  Inputs  to 

which  a function  can  be  applied  the  less  general  It  Is.  Thus,  this 
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measure  Is  based  on  the  number  of  modules  which  are  data  value 
limited.  This  can  be  determined  at  design  and  Impimentatlon. 

(5)  All  constants  should  be  defined  once. 

This  element,  in  effect,  defines  a constant  as  a parametric  value. 
At  one  place  in  the  module  or  data  base  it  can  be  changed  to  accom- 
modate a different  application  of  the  function  of  that  module, 
e.g. , to  calculate  a mathematical  relationship  at  a greater  degree 
of  precison  or  to  represent  the  constant  of  gravitation  of  a 
different  planet  than  earth,  etc.  Thus,  if  this  rule  Is  fol- 
lowed, the  effort  required  to  apply  the  module  In  a different 
environment  is  smaller.  The  measure  is  based  on  the  number  of 
modules  which  violate  this  concept  during  design  and 
implementation. 

6. 2. 2. 9 Expandability 

EX.l  Data  Storage  Expansion  Measure  (design  and  implmentation  phase  at 
system  level).  The  metric  is  the  sum  of  the  scores  of  the  following  appli- 
cable elements  divided  by  the  number  of  applicable  elements. 

(1)  Logical  processing  independent  of  storage  specification/require- 
ments. The  logical  processing  of  a module  is  to  be  independent 
of  storage  size,  buffer  space,  or  array  sizes.  The  design  pro- 
vides for  variable  dimensions  and  dynamic  array  sizes  to  be  defined 
parametrically.  The  metric  is  based  on  the  number  of  modules  con- 
taining hard-coded  dimensions  which  do  not  exemplify  this  concept. 

(2)  Percent  of  memory  capacity  uncommitted  (implementation  only). 

The  amount  of  memory  available  for  expansion  is  an  important  mea- 
sure. This  measure  identifies  the  percent  of  available  memory 
which  has  not  been  utilized  In  implementing  the  current  system. 

EX. 2 Extensibility  Measure  (design  and  Implementation  phases  at  the  system 
level).  The  metric  Is  the  sum  of  the  scores  of  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 
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(1)  Accuracy,  convergence,  timing  attributes  which  control  processing 
are  parametric. 

A module  which  can  provide  varying  degrees  of  convergence  or  timing 
to  achieve  greater  precision  provides  this  attribute  of  extensibil- 
ity. Hard-coded  contrui  parameters,  counters,  clock  values,  etc. 
violate  this  measure.  This  measure  Is  based  on  the  number  of  mod- 
ules which  do  not  exemplify  this  characteristic.  A determination 
can  be  made  during  design  and  Implementation. 

(2)  Modules  table  driven. 

The  use  of  tables  within  a module  facilitates  different  representa- 
tions and  processing  characteristics.  This  measure  which  can  be 
applied  during  design  and  Implementation  Is  based  on  the  number  of 
modules  which  are  not  table  driven. 

(3)  Percent  of  speed  capacity  uncoimil tted  (Implementation  only). 

A certain  function  may  be  required  In  the  performance  requirements 
specification  to  be  accomplished  In  a specified  time  for  overar 
timing  objectives.  The  amount  of  time  not  used  by  the  current 
Implementation  of  the  function  is  processing  time  av  ble  for 
potential  expansion  of  computational  capabilities.  sure 

Identifies  the  percent  of  total  processing  time  tha' 
uncommitted. 

6.2.2.10  Instrumentation 

IN.l  Module  testing  measure  (design  and  Implementation  phases,  first  at  mod- 
ule level  then  system  level).  The  system  level  metric  Is  an  average  of  a11 
module  measures.  The  module  measure  Is  the  average  score  of  the  following 
two  elements: 

(1)  Path  coverage. 

Plans  for  testing  the  various  paths  v.  thin  a module  should  be  made 
during  design  and  the  test  cases  actually  developed  during  imple- 
mentation. This  measure  Identifies  the  number  of  paths  planned  to 
be  tested  divided  by  the  total  number  of  paths. 

(2)  Input  parameters  boundary  tested. 

The  other  aspect  of  mudule  testing  Involves  testing  the  Input 


ranges  to  the  module.  This  Is  done  by  exercising  the  module  at  the 
various  boundary  values  of  the  Input  parameters.  Plans  to  do  this 
must  be  specified  during  design  and  coded  during  Implementation. 

The  measure  Is  the  number  of  parameters  to  be  boundary  tested 
divided  by  the  total  number  of  parameters. 

IN. 2 Integration  Testing  Measure  (design  and  Implementation  phases  at  system 
level).  The  metric  1s  the  averaged  score  of  the  following  two  elements. 

(1)  Module  interfaces  tested. 

One  aspect  of  Integration  testing  Is  the  testing  of  all  module  to 
module  Interfaces.  Plans  to  accomplish  this  testing  are  prepared 
during  design  and  the  tests  are  developed  during  implementation. 

The  measure  is  based  on  the  number  of  Interfaces  to  be  tested 
divided  by  the  total  number  of  Interfaces. 

(2)  Performance  requirements  (timing  and  storage)  coverage. 

The  second  aspect  of  Integration  testing  Involves  checking  for  com- 
pliance at  the  module  and  subsystem  level  with  the  performance 
requirements.  This  testing  Is  planned  during  design  and  the  tests 
are  developed  during  implementation.  The  measure  Is  the  number 
of  perfonnance  requirements  to  be  tested  divided  by  the  total 
nunber  of  performance  requirements. 


IN. 3 System  Testing  Measure  (design  and  Implementation  phases  at  the  system 
level).  The  metric  Is  the  averaged  score  of  the  turn  elements  below. 


(1)  Module  Coverage. 

One  aspect  of  system  testing  which  can  be  measured  as  early  as  the 
design  phase  Is  the  equivalent  to  path  coverage  at  the  module  level. 
For  all  system  test  scenarios  planned,  the  percent  of  all  of  the 
modules  to  be  exercised  Is  Important. 

(2)  Identification  of  test  Inputs  and  outputs  In  sutnnary  form. 

The  results  of  tests  and  the  manner  In  which  these  results  are 
displayed  are  very  Important  to  the  effectiveness  of  testing.  This 
is  especially  true  during  system  testing  because  of  the  potentially 
large  volume  of  Input  and  output  data.  This  measure  simply  identi- 
fies If  the  capability  exists  to  display  test  Inputs  and  outputs 

In  a summary  fashion.  The  measure  can  be  applied  to  the  plans 
and  specifications  in  the  design  phase  and  the  development  of 
this  capability  during  Implementation. 

6.2.2.11  Self  Descriptiveness 

SD.l  Quantity  of  Comments  (Implementation  phase  at  module  level  first  and 
then  system  level).  The  metric  Is  the  number  of  comment  lines  divided  by  the 
total  number  of  lines  in  each  module.  Blank  lines  are  not  counted.  The 
average  value  is  computed  for  the  system  level  metric. 


SO. 2 Effectiveness  of  Comments  Measure  (Implementation  phase  at  system  level). 
The  metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  Modules  have  standard  formatted  prologue  comments. 

The  Items  to  be  contained  In  the  prologue  comments  are  listed  In 
Table  6.2-1.  This  Information  Is  extremely  valuable  to  new 
personnel  who  have  to  work  with  the  software  after  development, 
performing  maintenance,  testing,  changes,  etc.  The  measure  at 
the  system  level  Is  based  on  the  number  of  modules  which  do  not 
comply  with  a standard  format  or  do  not  provide  complete  Information. 


(2)  Coirments  set  off  from  code  In  uniform  manner. 

Blank  lines,  bordering  asterisks,  specific  card  columns  are  some  of 
the  techniques  utilized  to  aid  In  the  Identification  of  comments. 

The  measure  Is  based  on  the  number  of  modules  which  do  not  follow 
the  conventions  established  for  setting  off  the  comments. 

(3)  All  transfers  of  control  and  destinations  comnented. 

This  form  of  comment  aids  In  the  understanding  and  ability  to  follow 
the  logic  of  the  module.  The  measure  Is  based  on  the  number  of 
modules  which  do  not  comply. 

(4)  All  machine  dependent  code  comnented. 

Comments  associated  with  machine  dependent  code  are  Important  not 
only  to  explain  what  Is  being  done  but  also  serves  to  Identify 
that  portion  of  the  module  as  machine  dependent.  The  metric  Is 
based  on  the  number  of  modules  which  do  not  have  the  machine 
dependent  code  commented. 

(5)  All  non-standard  HOL  statements  comnented. 

A similar  explanation  to  (4)  above  Is  applicable  here. 
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(6)  Attributes  Of  all  declared  variables  comnented. 

The  usage*  properties*  units*  etc.*  of  variables  are  to  be  explained 
In  comments.  The  measure  Is  based  on  the  number  of  modules  which  do 
not  follow  this  practice. 

(7)  Comments  do  not  just  repeat  operation  described  In  language. 

Conments  are  to  describe  why  not  what.  A comment*  Increment  A by  1, 
for  the  statement  A«A^1  provides  no  new  Information.  A comment, 
Increment  the  table  look-up  Index,  Is  more  valuable  for  under- 
standing the  logic  of  the  module.  The  measure  Is  based  on  the 
number  of  modules  In  which  conments  do  not  explain  the  why's. 

SO. 3 Descriptiveness  of  Implementation  Language  Measure  (Implementation 
phase  at  system  level).  The  metric  Is  the  sum  of  the  scores  of  the  following 
applicable  elements  divided  by  the  number  of  applicable  elements. 

(1)  High  Order  Language  used. 

An  HOL  Is  much  more  self-descriptive  than  assembly  language.  The 
measure  Is  based  on  the  number  of  modules  which  are  Implemented, 

In  whole  or  part.  In  assembly  or  machine  language. 

(2)  Standard  format  for  organization  of  modules  followed. 

A specific  format  ordering  such  as  prologue  conments*  declarative 
statements,  executable  statements  is  to  be  uniformly  used  In 
modules.  This  measure  Is  based  on  the  number  of  modules  which  do 
not  comply  with  the  standard  format  established. 

(3)  Variable  names  (mnemonics)  descriptive  of  physical  or  functional 
property  represented. 

While  the  metric  appears  very  subjective.  It  Is  quite  easy  to 
Identify  If  variable  names  have  been  chosen  with  self-descriptive- 
ness In  mind.  Three  variable  names  such  as  NAME,  POSIT,  SALRY 
are  far  better  and  more  easily  recognized  as  better  than  Al,  A2, 

A3.  The  measure  Is  based  on  the  number  of  modules  vhich  do  not 
utilize  descriptive  names. 


(4)  Source  code  logically  blocked  and  Indented. 

Techniques  such  as  blocking,  paragraphing.  Indenting  for  specific 
constructs  are  well  established  and  are  to  be  followed  uniformly 
within  a system.  This  measure  Is  based  on  the  number  of  modules 
which  do  not  comply  with  a uniform  technique. 

(5)  One  statement  per  line. 

The  use  of  continuation  statements  and  multiple  statements  per  line 
causes  difficulty  in  reading  the  code.  The  measure  Is  the  number 
of  continuations  plus  the  number  of  multiple  statement  lines  divided 
by  the  total  number  of  lines  for  each  module  and  then  averaged  over 
all  of  the  modules  In  the  system. 

(6)  No  language  keywords  used  as  names. 

Some  languages  allow  keywords  to  be  used  as  statement  labels  or  as 
variables.  This  practice  Is  confusing  to  a reader.  The  measure  Is 
based  on  the  number  of  modules  1n  which  a keyword  Is  used  In  this 
manner. 

6.2.2.12  Execution  Efficiency 

EE.l  Performance  Requirements  allocated  to  design  (design  phase  at  system 
level).  Performance  requirements  for  the  system  must  be  broken  down  and 
allocated  appropriately  to  the  modules  during  the  deslgi.  This  metric  simply 
Identifies  If  the  performance  requirements  have  (1)  or  have  not  (0)  been 
allocated  during  the  design. 

EE. 2 Iterative  Processing  Efficiency  Measure  (design  and  Implementation 
phases  at  module  level  first).  The  metric  at  the  module  level  Is  the  sum  of 
the  scores  of  the  following  applicable  elements  divided  by  the  number  of 
elements.  At  the  system  level  It  Is  an  averaged  score  for  all  of  the  modules. 

(1)  Non-loop  dependent  computations  kept  out  of  loop. 

Such  practices  as  evaluating  constants  In  a loop  are  to  be  avoided. 
This  measure  Is  based  on  the  number  of  non-loop  dependent  statements 


detailed  design  representation  during  design  and  from  the  code 
during  Implementation. 

(2)  Performance  Optimizing  CompHer/Assembly  language  used  (Implementation 
only). 

This  Is  a binary  measure  which  identifies  If  a performance  optimizing 
compiler  was  used  (1)  or  If  assembly  language  was  used  to  accomplish 
performance  optimization  (1)  or  not  (0). 

(3)  Compound  expressions  defined  once  (Implementation  only). 

Repeated  compound  expressions  are  to  be  avoided  from  an  efficiency 
standpoint.  This  metric  Is  based  on  the  number  of  compound 
expressions  which  appear  more  than  once. 

(4)  Number  of  overlays. 

The  use  of  overlays  requires  overhead  as  far  as  proces^ng  time. 

This  measure,  the  Inverse  of  the  number  of  overlays,  reflects  that 
overhead.  It  can  be  applied  during  design  when  the  ov^lay  scheme 
1$  defined  and  during  Implementation. 

(5)  Free  of  bit/byte  packing/ unpacking  In  loops. 

This  Is  a binary  measure  Indicating  the  overhead  Involved  In  bit/byte 
packing  and  unpacking.  Placing  these  activities  within  loops  should 
be  avoided  If  possible. 

(6)  Free  of  nonfunctional  executable  code  (Implementation  only). 

Segments  of  executable  code  which  do  not  perform  a relevant  function 
are  obvious  Inefficiencies.  They  arise  most  often  during  redesign 
or  editing  when  updates  are  made  without  complete  removal  of  obsolete 
code.  This  element  can  be  measured  at  Implementation  only  and  Is 
based  on  the  number  of  nonfunctional,  yet  executable  lines  of  code. 
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(7)  Decision  Statements  efficiently  coded  (Implementation  only). 

This  measure  Is  based  on  the  number  of  Inefficiently  coded  decision 
statements  divided  by  the  total  number  of  decision  statements.  An 
example  of  an  Inefficiently  coded  decision  statement  Is  not  having 
the  most  frequently  exercised  alternative  of  an  IF  statement  be  the 
THEN  clause. 

(8)  Module  linkages  (Implementation  only,  requires  execution). 

This  measure  essentially  represents  the  1nter-modu1e  comnunl cation 
overhead.  The  measure  Is  based  on  the  amount  of  execution  time 
spent  during  module  to  module  comnunl cat Ion. 

(9)  Operating  System  linkages  (Implementation  only,  requires  execution). 
This  measure  represents  the  module  to  OS  communication  overhead. 

The  measure  Is  based  on  the  amount  of  execution  time  spent  during 
module  to  OS  communications. 

EE. 3 Data  Usage  Efficiency  Measure  (design  and  Implementation  phases  applied 
at  module  level  first).  The  metric  at  the  module  level  Is  the  sum  of  the 
scores  of  the  following  applicable  elements  divided  by  the  number  of  applicable 
elements.  The  system  metric  Is  the  averaged  value  of  all  of  the  module  metric 
values. 

(1)  Data  grouped  for  efficient  processing. 

The  data  utilized  by  any  module  Is  to  be  organized  In  the  data  base, 
buffers,  arrays,  etc.,  in  a manner  which  facilitates  efficient 
processing.  The  data  organization  during  design  and  implementation  Is 
to  be  examined  to  provide  this  binary  measure. 

(2)  Variables  Initialized  when  declared  (Implementation  only). 

This  measure  Is  based  on  the  number  of  variables  used  In  a module 
which  are  not  Initialized  when  declared. 
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Efficiency  is  lost  ithen  variables  are  initialized  during  execution 
of  a function  or  repeatedly  initialized  during  iterative  processing. 

(3)  No  mix-mode  expressions  (implementation  only). 

Processing  overhead  is  consumed  by  mix-mode  expressions  which  are 
otherwise  unnecessary.  This  measure  is  based  on  the  number  of  mix- 
mode expressions  found  In  a module. 

(4)  Comon  choice  of  units/types. 

For  similar  reasons  as  expressed  above  (3)  this  convention  is  to  be 
followed.  The  measure  is  the  Inverse  of  the  number  of  operations 
performed  which  have  uncommon  units  or  data  types. 

(5)  Data  indexed  or  referenced  for  efficient  processing. 

Not  only  the  data  organization.  (1)  above,  but  the  linkage  scheme 
between  data  items  effects  the  processing  efficiently.  This  is  a 
binary  measure  of  whether  the  indexing  utilized  for  the  data  was 
chosen  to  facilitate  processing. 

6.2.2.13  Storage  Efficiency 

SE.l  Storage  Efficiency  Measure  (design  and  implementation  phases  at  module 
level  first  then  system  level).  The  metric  at  the  module  level  is  the  sum  of 
the  scores  of  the  following  applicable  elements  divided  by  the  number  of 
applicable  elements.  The  metric  at  the  system  level  is  the  averaged  value  of 
all  of  the  module  metric  values. 

(1)  Storage  Requirements  allocated  to  design  (design  phase  only). 

The  storage  requirements  for  the  system  are  to  be  allocated  to  the 
Individual  modules  during  design.  This  measure  is  a binary  measure 
of  whether  that  allocation  is  explicitly  made  (1)  or  not  (0). 


(2)  Virtual  Storage  Facilities  Used. 

The  use  of  virtual  storage  or  paging  techniques  enhances  the 
storage  efficiency  of  a system.  This  Is  a binary  measure  of  whether 
these  techniques  are  planned  for  and  used  (1)  or  not  (0). 

(3)  Common  data  defined  only  once  (Implementation  only). 

Often,  global  data  or  data  used  comnonly  are  defined  more  than 

once.  This  consumes  storage.  This  measure  Is  based  on  the  ntanber  i 

of  variables  that  are  defined  In  a module  that  have  been  defined 

elsewhere.  i 

i 

(4)  Program  Segmentation. 

Efficient  segmentation  schemes  minimize  the  maximum  segment  length 
to  minimize  the  storage  requirement.  This  measure  Is  based  on 
the  maximum  segment  length.  It  Is  to  be  applied  during  design  when 
estimates  are  available  and  during  Implementation. 

(5)  Data  Segmentation. 

The  amount  of  data  referenced  by  a module  In  the  form  of  arrays. 

Input  buffers,  or  global  data,  often  Is  small  compared  to  the 
size  of  the  storage  areas  required.  This  represents  an  Ineffi- 
cient use  of  storage.  The  measure  Is  based  on  the  amount  of  un- 
used data  divided  by  the  total  amount  of  data  available  to  a 
module. 

(6)  Dynamic  memory  management  used. 

This  Is  a binary  measure  emphasizing  the  advantages  of  using  dy- 
namic memory  management  techniques  to  minimize  the  amount  of 
storage  required  during  execution.  This  Is  planned  during  design 
and  used  during  1nq)lementat1on. 

(7)  Data  packing  used  (Implementation  only). 

While  data  packing  was  discouraged  In  EE. 2 (5)  In  loops  because  of 
the  overhead  It  adds  to  processing  time.  In  general  It  Is  bene- 
ficial from  a storage  efficiency  viewpoint.  This  binary  measure 
applied  during  Implmnentatlon  recognizes  this  fact. 
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(8)  Fre«  of  nonfunctional  code  (Inplenentatlon  only). 

Nonfunctional  code*  Mhether  executable  (see  EE. 2 (6))  or  not.  con- 
sumes storage  space  so  It  Is  undesirable.  This  measure  Is  based  on 
the  number  of  lines  of  code  which  are  nonfunctional. 

(9)  No  duplicate  code. 

Duplicate  code  should  be  avoided  for  the  same  reason  as  (6)  above. 

This  measure  which  Is  to  be  applied  during  design  and  Implementation  Is 
based  on  the  amount  of  duplicate  code. 

(10)  Storage  optimizing  compller/assembly  language  used  (Implementation 
only). 

This  binary  measure  Is  similar  to  EE. 2 (2)  except  from  the  viewpoint 
of  storage  optimization. 

(11)  Free  of  redundant  data  elements  (Implementation  only). 

This  measure  pertains  to  the  data  base  and  Is  based  on  the  number 
of  redundant  data  elements. 

6.2.2.14  Access  Control 

AC.l  Access  Control  Checklist  (all  three  phases  at  system  level). 

The  metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  the  number  of  applicable  elements. 

(1)  User  I/O  Access  controls  provided. 

Requirements  for  user  access  control  must  be  Identified  during  the 
requirements  phase.  Provisions  for  Identification  and  password 
checking  must  be  designed  and  Implemented  to  coiqply  with  the  require- 
ments. This  binary  measure  applied  at  all  three  phases  Identifies 
whether  attention  has  been  placed  on  this  area. 

(2)  Data  Base  Access  controls  provided. 

This  binary  measure  Identifies  whether  requirements  for  data  base 
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controls  have  been  specified,  designed  and  the  capabilities  Imple- 
mentated.  Examples  of  data  base  access  controls  are  authorization 
tables  and  privacy  locks. 


1 
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(3)  Memory  protection  across  tasks. 

Similar  to  (1)  and  (2)  above,  this  measure  Identifies  the  progression 
from  a requirements  statement  to  Implementation  of  memory  protection 
across  tasks.  Examples  of  this  type  of  protection,  often  times  pro- 
vided to  some  degree  by  the  operating  system,  are  preventing  tasks  from 
Invoking  other  tasks,  tasks  from  accessing  system  registers,  and  the 
use  of  privileged  conmands. 

6.2.2.15  Access  Audit 

AA.l  Access  Audit  Checklist  (all  three  phases  at  system  level). 

The  metric  Is  the  averaged  score  of  the  following  two  elements. 

(1)  Provisions  for  recording  and  reporting  access. 

A statement  of  the  requirement  for  this  type  capability  must  exist  In 
the  requirements  specification.  It  Is  to  be  considered  In  the  design 
specification,  and  coded  during  Implementation.  This  binary  metric 
applied  at  all  three  phases  Identifies  whether  these  steps  are 
being  taken.  Examples  of  the  provisions  which  might  be  considered 
would  be  the  recording  of  terminal  linkages,  data  file  accesses, 
and  jobs  run  by  user  Identification  and  time. 

(2)  Provisions  for  Immediate  Indication  of  access  violation. 

In  addition  to  (1)  above,  access  audit  capabilities  required 
might  Include  not  only  recording  accesses  but  Immediate  Identifica- 
tion of  unauthorized  accesses,  whether  Intentional  or  not.  This 
measure  traces  the  requirement,  design,  and  Implementation  of 
provisions  for  this  capability. 
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6.2.2.16  Operability 

OP.l  Operability  Checklist  (all  three  phases  at  system  level). 

The  metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements 
divided  by  tiie  number  of  applicable  elements. 

(1)  All  steps  of  operation  described. 

This  binary  measure  applied  at  all  three  phases  Identifies  whether 
the  operating  characteristics  have  been  described  In  the  require- 
ments specification,  and  If  this  description  has  been  transferred 
Into  an  Implementable  description  of  the  operation  (usually  In  an 
operator's  manual).  The  description  of  the  operation  should  cover 
the  normal  sequential  steps  and  all  alternative  steps. 

(2)  All  error  conditions  and  responses  appropriately  described  to 
operator. 

The  requirement  for  this  capability  must  appear  In  the  requirements 
specification,  must  be  considered  during  design,  and  coded  during 
Implementation.  Error  conditions  must  be  clearly  identified  by 
the  system.  Legal  responses  for  all  conditions  are  to  be  either 
documented  and/or  prompted  by  the  system.  This  Is  a binary  mea- 
sure to  trace  the  evolution  and  Implementation  of  these  capabilities. 

(3)  Provisions  for  operator  to  Interrupt,  obtain  status,  save,  modify, 
and  continue  processing. 

The  capabilities  provided  to  the  operator  must  be  considered  during 
the  requirements  phase  and  then  designed  and  Implemented.  Examples 
of  operator  capabilities  Include  halt/resume  and  check  pointing. 

This  Is  a binary  measure  to  trace  the  evolution  of  these 
capabilities. 

(4)  Number  of  operator  actions  reasonable  (Implementation  only,  re- 
quires execution). 

The  number  of  operator  errors  can  be  related  directly  to  the  mimber 
of  actions  required  during  a time  period.  This  measure  Is  based  on 
the  amount  of  time  spent  requiring  manual  operator  actions  divided 
by  the  total  time  required  for  the  Job. 


6-62 


(5)  Job  set  up  and  tear  down  procedures  described  (Implementation  only). 
The  specific  tasks  Involved  In  setting  up  a job  and  completing  It 
are  to  be  described.  This  Is  usually  documented  during  the  imple- 
mentation phase  when  the  final  version  of  the  system  Is  fixed. 

This  Is  a binary  measure  of  the  existence  of  that  description. 

(6)  Hard  copy  log  of  Interactions  maintained  (design  and  Implementation 
phases). 

This  Is  a capability  that  must  be  planned  for  In  design  and  coded 
during  Implementation.  It  assists  In  correcting  operational  errors. 
Improving  efficiency  of  operation,  etc.  This  measure  Identifies 
whether  it  Is  considered  In  the  design  and  Implementation  phases  (1) 
or  not  (0). 

(7)  Operator  messages  consistent  and  responses  standard  (design  and 
implementation  phases). 

This  Is  a binary  measure  applied  during  design  and  Implementation  to 
Insure  that  the  Interactions  between  the  operator  and  the  system  are 
simple  and  consistent.  Operator  responses  such  as  YES,  NO,  GO,  STOP, 
are  concise,  simple,  and  can  be  consistently  used  throughout  a system. 
Lengthy,  differently  formated  responses  not  only  provide  difficulty 
to  the  operator  but  also  require  complex  error  checking  routines. 

6.2.2.17  Training 

TG.l  Training  Checklist  (design  and  Implementation  at  system  level).  The 
metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements  divided  by 
the  number  of  applicable  elements. 

(1)  Lesson  Plans/Training  Material  developed  for  operators,  end  users, 
roaintalners  (Implementation  phase  only). 

This  Is  a binary  measure  of  whether  this  type  documentation  Is 
provided  during  the  implementation  phase. 
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(2)  keallstlc  simulated  exercises  provided  (Implementation  only). 

This  Is  a binary  measure  of  Mhether  exercises  which  represent  the 
operational  environment,  are  developed  during  the  Implementation 
phase  for  use  In  training. 

(3)  Sufficient  'help'  and  diagnostic  Information  available  on-line. 

This  Is  a binary  measure  of  whether  the  capability  to  aid  the 
operator  In  familiarization  with  the  system  has  been  designed  and 
built  Into  the  system.  Provision  of  a list  of  legal  coimands  or  a 
list  of  the  sequential  steps  Involved  In  a process  are  examples. 

6.2.2.18  Communicativeness 

CM.l  User  Input  Interface  fteasure  (all  three  phases  at  system  level). 

The  metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements  divi- 
ded by  the  number  of  applicable  elements. 

(1)  Default  values  defined  (design  and  Implementation). 

A method  of  minimizing  the  amount  of  Input  required  is  to  provide 
defaults.  This  measure,  applied  during  design  and  Implementation, 

Is  based  on  the  number  of  defaults  allowed  divided  by  the  total 
number  of  Input  parameters. 

(2)  Input  formats  uniform  (design  and  Implementation). 

The  greater  the  number  of  Input  formats  there  are  the  more  difficult 
the  system  is  to  use.  This  measure  Is  based  on  the  total  number  of 
Input  formats. 

(3)  Each  Input  record  self-identifying 

Input  records  which  have  self- Identifying  codes  enhance  the  accuracy 
of  user  Inputs.  This  measure  Is  based  on  the  number  of  Input 
records  that  are  not  self  Identi^ing  divided  by  the  total  number  of 
Input  records.  It  Is  to  be  applied  at  design  and  Implementation. 
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CM. 2 User  Output  Interface  Measure  (all  three  phases  at  system  level). 

The  metric  is  the  sum  of  the  scores  of  the  following  applicable  elements  divided 
by  the  number  of  applicable  elements. 

(1)  Selective  Output  Controls. 

The  existence  of  a requirement  for,  design  for,  and  implementation 
of  selective  output  controls  is  indicated  by  this  binary  measure. 
Selective  controls  include  choosing  specific  outputs,  output  formats, 
amount  of  output,  etc. 

(2)  Outputs  have  unique  descriptive  user  oriented  labels  (design  and 
implementation  only). 

This  is  a binary  measure  of  the  design  and  implementation  of  unique 
output  labels,  lii  addition,  the  labels. are  to  be  descriptive  to  the 
user.  This  includes  not  only  the  labels  which  are  used  to  reference 
an  output  report  but  also  the  title,  column  headings,  etc.  within  that 
report . 
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Input  can  be  verified  by  user  prior  to  execution  (design  and 
implementation). 

The  capability,  displaying  input  upon  request  or  echoing  the  input 
automatically,  enables  the  user  to  check  his  inputs  before 
processing.  This  is  a measure  of  the  existence  of  the  design  and 
implementation  of  this  capability. 

Input  terminated  by  explicitly  defined  logical  end  of  input  (design 
and  implementation). 

The  user  should  not  have  to  provide  a count  of  input  cards.  This  is 
a binary  measure  of  the  design  and  implementation  of  this  capability. 

Provision  for  specifying  input  from  different  media. 

The  flexibility  of  input  must  be  decided  during  the  requirements 
analysis  phase  and  followed  through  during  design  and  implementation. 
This  is  a binary  measure  of  the  existence  of  the  consideration 
of  this  capability  during  all  three  phases. 
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(3)  Outputs  have  user  oriented  units  (design  and  Implementation). 

This  Is  a binary  measure  which  extends  (2)  above  to  the  Individual 
output  Items. 

(4)  Uniform  output  labels  (design  and  Implementation). 

This  measure  corresponds  to  CM.l  (2)  above  and  It  the  Inverse  of 

the  number  of  different  output  formats.  | 

(5)  Logical  groups  of  output  separated  for  user  examination  (design 
and  Implementation). 

Utilization  of  top  of  page,  blank  lines,  lines  of  asterisks,  etc., 
provide  for  easy  identification  of  logically  grouped  output.  This 
binary  measure  Identifies  If  these  techniques  are  used  during  design 
and  Implementation. 

, ii 

(6)  Relationship  between  error  messages  and  outputs  1s  unambiguous 
(design  and  Implementation). 

This  Is  a binary  measure  applied  during  design  and  Implementation 
which  Identifies  if  error  messages  will  be  directly  related  to  the 
output. 

(7)  Provision  for  redirecting  output  to  different  media. 

This  Is  a binary  metric  which  Identifies  If  consideration  Is  given 
to  the  capability  to  redirect  output  to  different  media  during 
requirements  analysis,  design,  and  Implementation. 


6.2.2.19  Software  System  Independence 

SS.l  Software  System  Independence  Measure  (design  and  Implementation  phases 
at  system  level).  The  metric  Is  the  sum  of  the  scores  of  the  following  appllc 
able  elements  divided  by  the  number  of  applicable  elements. 


• (1)  Dependence  on  Software  System  Utility  programs. 

The  more  utility  programs  that  are  used  within  a system  the  more 

I 

dependent  the  system  Is  on  that  software  system  environment.  A 

I < 
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SORT  utility  in  one  operating  system  is  unlikely  to  be  exactly 
similar  to  a SORT  utility  in  another.  This  measure  is  based  on 
the  number  of  programs  used  that  are  utility  programs  divided  by 
the  total  number  of  programs  in  the  system.  It  is  to  be  applied 
during  design  and  implementation. 

(2)  Dependence  on  Software  System  Library  Routines. 

For  similar  reasons  as  (1)  above  an  integration  function  provided 
by  one  operating  system  may  not  be  exactly  the  same  as  an  integra- 
tion function  provided  by  another  system.  Thus  the  more  library 
routines  used  the  more  dependent  the  system  is  on  its  current 
software  system  environment.  This  measure,  applied  at  design  and 
implementation,  is  based  on  the  number  of  library  routines  used 
divided  by  the  total  number  of  modules  in  the  system. 

(3)  Common,  standard  subset  of  language  used. 

The  use  of  nonstandard  constructs  of  a language  that  may  be  available 
from  certain  compilers  cause  conversion  problems  when  the  software  is 
moved  to  a new  software  system  environment.  This  measure  represents 
that  situation.  It  is  based  on  the  number  of  modules  which  are 
coded  in  a non-standard  subset  of  the  language.  The  standard  sub- 
set of  the  language  is  to  be  established  during  design  and  adhered 
to  during  implementation. 

(4)  Free  from  Operating  System  References. 

This  measure  is  based  on  the  number  of  modules  which  contain  calls 
to  the  operating  system.  While  (1)  and  (2)  above  identify  the 
number  of  support- type  programs  and  routines  which  might  have  to  be 
recoded  if  a change  in  software  system  environments  took  place, 
this  measure  identifies  the  percent  of  application  oriented  modules 
which  would  probably  have  to  be  changed.  The  metric  is  to  be  applied 
during  design  and  implementation. 
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6.2.2.20  Hachine  Independence 

MI.l  Machine  Independence  Measure  (design  and  Implementation  at  system  level). 

The  metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements  i 

divided  by  the  number  of  applicable  elements. 

i 

(1)  Progrannlng  language  used  available  on  other  machines. 

This  Is  a binary  measure  Identifying  If  the  programming  language 
used  Is  available  (1)  on  other  machines  or  not  (0).  This  means 

the  same  version  and  dialect  of  the  language.  | 

'! 

■1 

.1 

(2)  Free  from  Input/ output  references. 

Input  and  output  references  bind  a module  to  the  current  machine  con- 
figuration. Thus  the  fewer  modules  within  a system  that  contain 
Input  and  output  references,  the  more  localized  the  problem  becomes 
when  conversion  Is  considered.  This  measure  represents  that  fact 
and  Is  based  on  the  number  of  modules  within  the  system  that  contain 
1/0  references.  It  Is  to  be  applied  during  design  and  Implementation. 

(3)  Code  Is  Independent  of  word  and  character  size  (Implementation  only). 

Instructions  or  operations  which  are  dependent  on  the  word  or 
character  size  of  the  machine  are  to  be  either  avoided  or  param- 
etric to  facilitate  use  on  another  machine.  This  measure  applied  | 

to  the  source  during  Implementation  Is  based  on  the  number  of  < 

modules  which  contain  violations  to  the  concept  of  Independence  of  i 

word  and  character  size.  | 

\ 

(4)  Data  representation  machine  Independent  (Implementation  only).  | 

The  naming  conventions  (length)  used  are  to  be  standard  or  com-  I 

patible  with  other  machines.  This  measure  Is  based  on  the  nunt>er  ] 

of  modules  which  contain  variables  which  do  not  conform  to  standard  j 

data  representations.  ] 


3 
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6.2.2.21  Coramunications  Cownonality 

CC.l  Communications  Commonality  Checklist  (all  three  phases  at  system 
level).  The  metric  Is  the  sum  of  the  scores  of  the  following  applicable 
elements  divided  by  the  number  of  applicable  elements. 

(1)  Definitive  statement  of  requirements  for  communcatlon  with  other 
systems  (requirements  only). 

During  the  requirement  phase,  the  comnunl cation  requirements 
with  other  systems  must  be  considered.  This  Is  a binary  measure  of 
the  existence  of  this  consideration. 

(2)  Protocol  standards  established  and  followed. 

The  communcatlon  protocol  standards  for  communication  with  other 
systems  are  to  be  established  during  the  design  phase  and  followed 
during  implementation.  This  binary  measure  applied  at  each  of 
these  phases  Indicates  whether  the  standards  were  established  and 
followed. 

(3)  Single  module  Interface  for  Input  from  another  system. 

The  more  modules  which  handle  Input  the  more  difficult  It  Is  to 
Interface  with  another  system  and  Implement  standard  protocols. 

This  measure  based  on  the  Inverse  of  the  number  of  modules  which 
handle  Input  Is  to  be  applied  to  the  design  specification  and  source 
code. 


(4)  Single  module  Interface  for  output  to  another  system 

For  similar  reasons  as  (3)  above  this  measure  Is  the  Inverse  of 
the  number  of  output  modules. 

6.2.2.22  Data  Commonality 

DC.l  Data  Commonality  Checklist  (all  three  phases  at  system  level).  The 
metric  Is  the  sum  of  the  scores  of  the  following  applicable  elements  divided 
by  the  number  of  applicable  elements. 
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(1)  Definitive  statement  for  standard  data  representation  for  coiiinun1ca> 
tions  with  other  systems  (requirements  only). 

This  Is  a binary  measure  of  the  existence  of  consideration  for 
standard  data  representation  between  systems  which  are  to  be  Interfaced. 
This  must  be  addressed  and  measured  In  the  requirements  phase. 

• 

(2)  Translation  standards  among  representations  established  and  followed 
(design  and  Implementation). 

More  than  one  translation  from  the  standard  data  representations  used 
for  Interfacing  with  other  systems  may  exist  within  a system.  Standards 
for  these  translations  are  to  be  established  and  followed.  This  binary 
measure  Identifies  If  the  standards  are  established  during  design  and 
followed  during  implementation. 


I 

I 


l 

I 


(3)  Single  module  to  perform  each  translation  (design  and  Implementation). 
For  similar  reasons  as  CC.l  (3)  and  (4)  above,  this  measure  Is  the 
Inverse  of  the  maximum  number  of  modules  which  perform  a translation. 

6.2.2.23  Conciseness 

CO.l  Halstead's  Measure  (Implementation  phase  at  module  level  first  then  system 
level).  The  metric  Is  based  on  Halstead's  concept  of  length  (FHALSMTT]). 

The  observed  length  of  a module  Is 
No  ■ Nj  + Ng  where: 

Nj  ■ total  usage  of  all  operators  In  a module 
N2  ■ total  usage  of  all  operators  In  a module 


The  calculated  length  of  a module  Is 

Nc  “ nilog2ni  + n2log2n2  where: 

nj  » number  of  unique  operators  In  a module 

02  ■ number  of  unique  operators  In  a module 


The  metric  Is  normalized  as  follows: 


1 - 
0 If 


or, 

greater  than  1 


At  a system  level  the  metric  Is  the  averaged  value  of  all  the  module  metric 
values. 
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6.3  SUWARIZATION  OF  METRICS 

Table  6.3-1  provides  siannary  figures  for  the  metrics  that  have  been 
established. 

41  metrics  have  been  established  for  the  11  factors  and  23  criteria.  These  41 
metrics  are  comprised  of  175  elements  or  specific  characteristics  of  a soft- 
ware product  that  can  be  measured  at  various  times  during  development  to  give 
an  indication  of  the  progression  toward  a desired  level  of  quality.  The 
metrics  are  applied  during  the  three  phases  of  development  as  shown.  Thus 
25  characteristics  of  the  software  product  are  measured  during  the  require- 
ments analysis  phase.  108  during  the  design  phase,  and  157  during 
implementation. 


Table  6.3-1  Sunmarlzatlon  of  Metrics 
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APPENDIX  A 

QUALITY  FACTORS  REFERENCES 
IN  THE  LITERATURE  WITH 
DEFINITIONS 


The  quality  factor  definitions  or  discussions  contained  in  this  appendix 
were  found  in  the  literature  or  generated  during  this  effort  to  provide 
additional  interpretations.  The  authors  of  the  quoted  or  paraphrased 
definitions  are  indicated  in  parentheses. 

We  wish  to  emphasize  that  any  errors,  omissions,  or  misleading  statements 
in  the  quoted  or  paraphrased  definitions  contained  herein  are  completely 
and  solely  our  responsibility  and  not  those  of  the  authors  referenced. 


PORTABILITY/TRANSFERABILITY 


programs  must  be  readily  transferable  among  different  equipment  configura- 
tions (Goodenough/Cul pepper) 

degree  of  transportability  Is  determined  by  number,  extent,  and  complexity 
of  changes,  and  hence  the  difficulty  In  Implementing  a software  processor 
which  can  mechanically  move  a program,  between  a specified  set  of  machines 
(Hague) 

machine  - Independence  (Marshall) 

measure  of  the  ease  of  moving  a computer  program  from  one  computing  environ- 
ment to  another  (Meahy,  Poole) 

how  quickly  and  cheaply  the  software  system  can  be  converted  to  perform 
the  same  functions  using  different  equipment  (Kosy) 

an  appropriate  environment  can  be  provided  on  most  computers  (Goos) 

extent  that  1t  can  be  operated  easily  and  well  on  computer  configurations 
other  than  Its  current  one  (Boehm) 

moving  software  from  one  computer  (environment)  to  another  (Lleblein) 

measure  of  the  effort  required  to  Install  a program  on  another  machine, 
another  operating  system,  or  a different  configuration  of  the  same 
machine  (Wulf) 

external  (black  box)  form,  fit,  and  function  characteristics  of  a program 
module  which  permit  its  use  as  a building  block  In  several  computer  ptograms 
(KSSC  PATHWAY) 

relates  to  how  quickly  and  easily  a software  system  can  be  transferred  from 
one  hardware  system  to  another  (USA  ISRAD) 

ease  of  conversion  of  a system  from  one  environment  to  another;  the 
relative  conversion  cost  fo'"  a given  conversion  method  (Glib) 

ACCEPTABILITY 

how  closely  the  ADP  system  meets  true  user  needs  (Kosy) 

measure  of  how  closely  the  computer  program  meets  the  true  needs  of  the 
user  (SAMSO) 

relates  to  degree  to  which  software  meets  the  user's  needs  Including  the 
clarity  and  unambiguity  of  the  specifications  and  the  effectiveness  of  the 
man-machine  Interface  (USA  ISRAD) 

does  the  software  meet  the  need  of  the  user  (Light) 

COMPLETENESS 

extent  to  which  software  fulfills  overall  mission  satisfaction  (McCall) 

extent  that  all  of  Its  parts  are  present  and  each  of  Its  parts  are  fully 
developed  (Boehm) 


CONSISTENCY 


degree  to  which  software  satisfies  specifications  (HcCall) 

extent  that  It  contains  uniform  notation,  terminology,  and  symbology  with- 
in Itself  and  the  extent  that  the  content  Is  traceable  to  the  requirements 
(Boehm) 


CORRECTNESS 

correctness  of  its  description  with  respect  to  the  objective  of  the  soft- 
ware as  specified  by  the  semantic  description  of  the  linguistic  level  It 
defines  (Dennis) 

the  coding  of  a computer  program  module  ^hlch  properly  and  completely 
implements  selected  overall  system  requirements  (NSSC  PATHWAY) 

relates  to  degree  to  which  software  is  free  from  design  and  program  defects 
(USA  ISRAD) 

the  program  Is  logically  correct  (Rubey) 


AVAILABILITY 


fraction  of  total  time  during  which  the  system  can  support  critical  func 
tions  (SADPR-85) 

error  recover  and  protection  (Liskov) 

probability  that  a system  is  operating  satisfactorily  at  any  point  in 
time,  when  used  under  stated  conditions  (Glib) 

RELIABILITY 


Includes  correctness,  testing  for  errors,  and  error  tolerance  (Ledgard) 

the  probability  that  the  software  will  satisfy  the  stated  operational 
requirements  for  a specified  time  interval  or  a jnit  application  in  the 
operational  environment  (JLC  SRWG) 

the  probability  that  a software  system  will  operate  without  failure  for 
at  least  a given  period  of  time  when  used  under  stated  conditions  (Kosy) 

extent  to  which  a program  can  be  expected  to  perform  its  intended  functions 
satisfactorily  (Thayer) 

ability  of  the  software  to  perform  its  functions  correctly  in  spite  of 
failures  of  computer  system  components  (Dennis) 

probability  that  a software  fault  does  not  occur  during  a specified  time 
Interval  (or  specified  number  of  software  operational  cycles)  which  causes 
deviation  from  required  output  by  more  than  specified  tolerances,  in  a 
specific  environment  (Thayer) 

measure  of  the  number  of  errors  encountered  In  a program  (layers) 
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RELIABILITY.  Continued 


probability  that  the  system  will  perform  satisfactorily  for  at  least  a 
given  time  interval,  when  used  under  stated  conditions  (Glib) 

.extent  to  which  a program  is  debugged,  can  get  whatever  degree  of 
reliability  one  is  willing  to  pay  for  (Casey) 

the  probability  that  the  computer  program  will  satisfactorily  execute 
for  at  least  a given  period  of  time  when  used  under  specific  conditions. 
(SAMSO) 

tasks  are  broken  into  easily  manageable  modules  and  programming  internal 
to  most  modules  remains  constant  (Whipple) 

the  degree  of  assurance  that  the  software  will  do  its  job  when  integrated 
with  the  hardware  (Light) 


ACCURACY 

where  mathematically  possible  a routine  should  give  an  approximation  that 
is  as  close  as  practicable  to  the  full  machine  precision  for  whatever 
machine  it  is  running  on  (Schonfelder) 

extent  that  its  outputs  are  sufficiently  precise  to  satisfy  its  intended 
use  (Boehm) 

the  mathematical  calculations  are  correctly  performed  (Rubey) 

measure  of  the  quality  of  freedom  from  error,  degree  of  exactness 
possessed  by  an  approximation  or  measurement  (Gilb) 


ROBUSTNESS 

routines  should  be  coded  so  that  when  it  is  not  possible  to  return  a result 
with  any  reasonable  accurracy  or  there  is  danger  of  causing  some  form  of 
machine  failure  they  should  detect  this  and  take  appropriate  actions 
(Schonfelder) 

quality  of  a program  that  determines  its  ability  to  continue  to  perform 
despite  some  violation  of  the  assumptions  in  its  specifications  (Wulf) 

program  should  test  input  for  plausability  and  validity  (Kernighan) 


EFFICIENCY 

measure  of  the  execution  behavior  of  a program  (execution  speed,  storage 
speed)  (Myers) 

execution  time,  storage  space,  # Instructions,  processing  time  (Kosy) 

extent  that  software  fulfills  its  purpose  without  waste  of  resources 
(Boehm) 

the  ratio  of  useful  work  performed  to  the  total  energy  expended  (Gilb) 
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EFFICIENCY,  Continued 


reduction  of  overall  cost  - run  time,  operation,  maintenance  (Kernighan) 
extremely  fast  run  time  and  efficient  overlay  capabilities  (Richards) 
computation  time  and  memory  usage  are  optimized  (Rubey) 

PERFORMANCE 

the  effectiveness  with  which  resources  of  the  host  system  are  utilized 
toward  meeting  the  objective  of  the  software  system  (Dennis) 

refers  to  such  attributes  as  size,  speed,  precision;  e.g.,  the  rate  at  which 
the  program  consumes  accountable  resources  (Wulf) 

CONCISENESS 

the  ability  to  satisfy  functional  requirements  with  e minimum  amount  of 
software  (McCall) 

extent  that  no  excessive  information  is  present  (Boehm) 

UNDERSTANDABILITY 

ease  with  which  the  implementation  can  be  understood  (Richards) 

reduced  comolexity,  reduced  redundancy,  clear  documentation/notation 
(Goodenough) 

extent  that  the  purpose  of  the  product  is  clear  to  the  evaluator  (Boehm) 

Documentation  remains  current  (Whipple) 
the  program's  intelligible  (Rubey) 

SELF-DESCRIPTIVENESS 

CLARITY 

measure  of  how  clear  a program  is,  i.e.  how  easy  it  is  to  read,  understand, 
and  use  (Ledgard) 

refers  to  the  ease  with  which  the  program  (and  its  documentation)  can  be 
understood  by  humans  (Wulf) 
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CLARITY,  Continued 

extent  that  It  contains  enough  information  for  a reader  to  determine  Its 
objectives,  assumptions,  constraints.  Inputs,  outputs,  components,  and 
status  (Boehm) 


LEGIBILITY 

extent  that  Its  functions  and  those  of  Its  components  statements  are 
easily  discerned  by  reading  the  code  (Boehm) 

MAINTAINABILITY 

measure  of  the  effort  and  time  required  to  fix  bugs  in  the  program 
(Myers) 

how  easy  it  Is  to  locate  and  correct  errors  found  In  operational  use 
(Kosy) 

extent  that  the  software  facilitates  updating  to  satisfy  new  requirements 
(Boehm) 

maintenance  Involves 

(1)  correction  to  heretofore  latent  bugs 

(2)  enhancements 

(3)  expansion 

(4)  major  redesign  (Lieblein) 

ease  with  which  a change  can  be  made  due  to 

(1)  bug  during  operation 

(2)  non-satisfaction  of  users  requirements 

(3)  changing  requirements 

(4)  obsolecence/upgrade  of  system  (McCall) 

probability  that  a failed  system  will  be  restored  to  operable  condition 
within  a specified  time  (611b) 


(McCall) 


STABILITY 

the  "ripple  effect"  or  how  many  modules  have  to  be  changed  when  you  make 
a change  (Myers) 

measure  of  the  lack  of  perclevable  change  in  a system  in  spite  of  the 
occurrence  In  the  environment  which  would  normally  be  expected  to 
cause  a change  (Glib) 

ADAPTABILITY 

how  much  time  and  effort  are  required  to  modify  a software  system  (Kosy) 

measure  of  the  ease  with  which  a program  can  be  altered  to  fit  differing 
user  Images  and  system  constraints  (Poole) 


ADAPTABILITY,  Continued 


measure  of  the  ease  of  extending  the  product,  such  as  adding  new  user 
functions  to  the  product  (Myers) 

a measure  of  the  effort  required  to  modify  a computer  program  to  add, 
change  or  remove  a function  or  use  the  computer  program  In  a different 
environment  (includes  concepts  of  flexibility  and  portability)  (SAMSO) 

relates  to  ability  of  software  to  operate  inspite  of  unexpected  input 
or  external  conditions  (USA  ISRAD) 


EXTENSIBILITY 

extent  to  which  system  can  support  extensions  of  critical  functions 
(SADPR-85) 


MODIFIABILITY 

measure  of  the  cost  of  changing  or  extending  the  program  (Myers) 

operational  experience  usually  shows  the  need  for  incremental  software 
improvements  (Goodenough) 

extent  that  it  facilitates  the  incorporation  of  changes,  once  the  nature 
of  the  desired  change  has  been  determined  (Boehm) 

quality  of  a program  that  reduces  the  effort  required  to  alter  it  in 
order  to  conform  to  a modification  of  its  specification  (Wulf) 

internal  (detailed  design)  characteristics  of  a program  module  are 
arranged  so  as  to  permit  easy  change  (NSSC  PATHWAY) 

use  of  HOL  reduces  programmer's  task  and  human  errors  and  allows  smaller 
units  to  be  tested  permitting  easier  de-bugging  (Whipple) 

the  program  is  easy  to  modify  (Rubey) 


ACCESSIBILITY 

extent  that  software  facilitates  the  selective  use  of  its  components 
(Boehm) 


FLEXIBILITY 


extent  to  which  system  can  absorb  workload  Increases  and  decreases  which 
require  modification  (SADPR-85) 

the  ability  of  a system  to  immediately  handle  different  logical  situations 
(Glib) 


FLEXIBILITY 
EXPARBRbIlTty 
ADagNTAftlllTY.  Continued 

how  easily  the  software  modules  comprising  the  system  or  subsystem 
can  be  rearranged  to  change  the  system's  functions  without  modifying 
any  of  the  modules  (Kosy) 

ease  of  changing,  expanding,  or  upgrading  a program  (Yourdon) 

the  software  modules  must  be  usable  in  a variety  of  contexts  (Culpepper) 

includes  changeability;  e.g.,  the  ease  of  correcting  bugs,  maintenance 
because  of  changing  specifications,  and  portability  to  move  to  another 
system  (Ledgard) 

extent  that  software  easily  accommodates  expansions  in  data  storage  require- 
ments or  component  computational  functions  (Boehm) 

attributes  of  software  which  allow  quick  response  to  changes  in  algorithms 
(Richards) 

- ability  to  reuse  the  software  and  transfer  it  to  another  processor  (includes 
reuse,  adaptability,  transferability  and  versatility  of  software)  (Light) 


INTEGRITY 

- how  much  the  operation  of  one  software  subsystem  can  protect  the  oper- 
ation of  another  (Kosy)  • 

a measure  of  the  degree  of  protection  the  computer  program  offers  against 
unauthorized  access  and  loss  due  to  controllable  events  (includes  the 
concepts  of  privacy  and  security)  (SAMSO)  i 

relates  to  ability  of  software  to  prevent  purposeful  or  accidental  | 

damage  to  the  data  or  software  (USA  ISRAD) 

ability  to  resist  faults  from  personnel,  the  security  problem,  or  from  \ 

the  environment,  a fault  tolerance  issue  (Light) 

probability  of  system  survival  when  subjected  to  an  attack  during  a time 
interval  (Gilb) 


' SECURITY 

i ' - the  ability  to  prevent  unauthorized  access  to  programs  or  data  (Kosy) 

( ' - extent  to  which  access  to  software,  data,  and  facilities  can  be  controlled 

I (SADPR-85) 

[•'  . ■ 

} i - measure  of  the  probability  that  one  system  user  can  accidentally  or 

; intentionally  reference  or  destroy  data  that  is  the  property  of  another 

[ user  or  inteiface  with  the  operation  of  the  system  (%ers) 

; ( 


SECURITY,  Continued 


I 


relates  to  the  ability  of  the  software  to  prevent  unauthorized  access  to 
the  system  or  system  elements  (USA  ISRAD) 


PRIVACY 

the  extent  to  which  access  to  sensitive  data  by  unauthorized  persons  can 
be  controlled  and  the  extent  to  which  the  use  of  the  data  once  accessed 
can  be  controlled  (McCall) 

relates  to  the  protection  level  for  data  in  the  system  and  the  individual's 
right  to  review  and  control  dissemination  of  data  (USA  ISRAD) 


USABILITY 

OPERABILITY 

measure  of  the  human  interface  of  the  program  (Myers) 

ease  of  operation  from  human  viewpoint,  covering  both  human  engineering 
and  ease  of  transition  from  current  operation  (SADPR-85) 

how  suitable  is  it  to  the  use  (Kosy) 

software  must  be  adequately  documented  so  that  it  can  be  easily  used 
and  maintained  (Culpepper) 

extent  that  it  is  convenient  and  practicable  to  use  (Boehm) 
the  program  is  easy  to  learn  and  use  (Rubey) 


\ 
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HUMAN  FACTORS 


every  program  presents  an  interface  to  its  human  users/operators,  by 
human  factors  we  refer  collectively  to  all  the  attributes  that  make  this 
interface  more  palatable:  ease  of  use,  error  protectedness,  quality  of 
documentation,  uniform  syntax,  etc.  (Wulf) 

extent  that  software  fulfills  its  purpose  without  wasting  user's  time 
and  energy  or  degrading  their  morale  (Boehm) 

measure  of  the  product's  ease  of  understanding,  ease  of  use,  difficulty 
of  misuse,  and  frequency  of  user's  errors  (Myers) 


COMMUNICATIVENESS 


extent  that  software  facilitates  the  specifications  of  inputs  and  provide 
outputs  whose  form  and  content  are  easy  to  assimilate  and  useful  (Boehm) 


ability  to  combine  arbitrary  program  modules  Into  larger  modules  without 
knowledge  of  the  construction  of  the  modules  (Goos) 

the  software  must  consist  of  modules  with  well  defined  Interfaces.  Inter- 
actions between  modules  must  occur  only  at  those  Interfaces  (Culpepper) 

extent  that  It  possesses  a definite  pattern  of  organization  of  Its 
Independent  parts  (Boehm) 

how  well  a program  Is  organized  around  Its  data  representation  and  flow 
of  control  (Kernighan) 

there  Is  no  Interference  between  program  entitles  (Rubey) 

formal  way  of  dividing  a program  Into  a number  of  sub  units  each  having 
a well  defined  function  and  relationship  to  the  rest  of  the  program 
(Mealy) 


UNIFORMITY 

module  should  be  usable  uniformly  (Goodenough) 


GENERALITY 

reusability 

measure  of  the  scope  of  the  functions  that  a program  performs  (layers) 

building  programs  from  reusable  software  modules  can  significantly  reduce 
production  costs  (Goodenough) 

how  broad  a class  of  similar  functions  the  system  can  perform  (Kosy) 

standardized  modules  can  be  lifted  from  one  program  and  used  In  another 
without  extensive  re-coding  or  re-testings  (Whipple) 

degree  to  which  a system  Is  appl1cable_1n  different  environments  (Glib) 


TESTABILITY 

Instrumentation  and  debugging  aids  (Liskov) 
minimize  testing  costs  (Yourdon) 

provision  of  facilities  In  the  design  of  programs  which  are  essential  to 
testing  complex  structures  (Edwards) 

extent  that  software  facilitates  the  establishment  of  acceptance  criteria 
and  supports  evaluation  of  Its  performance  (Boehm) 

a measure  of  the  effort  required  to  exercise  the  computer  program  to  see 
how  well  It  performs  In  a given  environment  and  If  It  activally  solves  the 
problem  It  was  supposed  to  solve  (SAMSO) 
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TESTABILITY,  Continued 


measure  of  our  ability  to  test  software  (Light) 


INTEROPERABILITY 


how  quickly  and  easily  one  software  system  can  be  coupled  t^  another  (Kosy) 


relates  to  how  quickly  and  easily  one  software  system  can  be  coupled  to 
another  (USA  ISRAO) 


CONVERTIBILITY 

degree  of  success  anticipated  in  readying  people,  machines,  and  procedures 
to  support  the  system  (SADPR-85) 


MANAGEABILITY 

degree  to  which  system  lends  itself  to  efficient  administration  of  its 
components  (SADPR-85) 


COST 


includes  not  only  development  cost,  but  also  the  costs  of  maintenance, 
training,  documentation,  etc.,  on  the  entire  life  cycle  of  the  program 
(Wulf) 

there  are  three  major  categories  of  cost: 

Economy  of  operation  - relates  to  cost  or  operating  system 

Economy  of  modification  - relates  to  cost  of  making  changes  to  software  to 
meet  new  requirements  or  correct  defects  resulting  in  errors  in  require- 
ments, design,  and  programming 

Economy  of  development  - relates  to  cost  of  entire  development  cycle  from 
identification  of  requirement  to  initial  operation 

development  and  maintenance  costs  (flyers) 

implementation  cost  and  operational  cost  (Gilb) 


I i , ACCOUNTABILITY 

4 

j ; - extent  that  code  usage  can  be  measured  (Boehm) 

I ! 
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SEIF-COKTAINEDNESS 

extent  to  which  a program  performs  all  Its  explicit  and  Implicit  functions 
within  Itself  (Boehm) 


EXPRESSION 

how  a program  Is  expressed  determines  In  large  measure  the  Intelligibility 
of  the  whole  (Kemighan) 


VALIDITY 

relates  to  degree  to  which  software  Implements  the  user's  specifications 
(USA  ISRAO) 


TIME 

two  major  categories  of  time: 

Modification  Time  - relates  to  total  elapse  time  from  point  when  new 
requirement  or  modification  Is  Identified  the  change  Is  Implemented 
and  validated 

Development  Time  - relates  to  total  elapsed  time  of  development 

(USA  ISRAO) 

development  time  (Myers) 

what  Is  the  expected  life  span  of  system  (Glib) 


COMPLEXITY 


TOLERANCE 


t 


I 
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measure  of  the  systems  ability  to  accept  different  forms  of  the  same 
Information  as  valid  or  withstand  a degree  of  variation  In  Input  with 
out  malfunction  or  rejection  (Glib) 


COMPATABILITY 

measure  of  portability  that  can  be  expected  of  systems  when  they  are 
moved  from  one  given  environment  to  another  (611b) 


REPAIRABILITY 

probability  that  a failed  system  will  be  restored  to  operable  condition 
within  a specified  active  repair  time  when  maintenance  Is  done  under 
specified  conditions  (Glib) 


SERVICEABILITY 


degree  of  ease  or  difficulty  with  which  a system  can  be  repaired  (Glib) 


APPENDIX  B 


DOCUMENTATION  CHARACTERISTICS 
SOFTWARE  SYSTEM  REQUIREMENTS  SPECIFICATION  AND  REVIEW 


This  document  describes  the  functional  requirements  and  capabilities  of  the 
software  system.  The  data  requirements,  essentially  from  the  view  point  of 
the  end  user,  are  described.  It  includes  the  operational  constraints  and 
considerations  imposed  on  the  software  by  the  hardware  system.  It  con- 
stitutes the  primary  interface  between  the  user/customer  and  the  developer, 
during  both  formal  and  informal  reviews.  As  such,  it  is  a primary  reference 
during  the  design  and  development  of  a software  system.  Because  of  this,  any 
revisions  are  continually  made  to  keep  it  as  current  as  possible. 

STANDARDS  AND  CONVENTIONS 


This  document  provides  guidelines  for  the  design  and  implementation  of 
computer  programs.  A standard  is  defined  to  be  a rule  which  must  be 
followed  to  produce  an  acceptable  product.  A convention  is  defined  to 
be  a recommended  procedure  which  will  enhance  the  quality  of  a program's 
operation  or  its  documentation.  Many  of  the  checklist  type  metrics 
described  are  related  to  standards  and  conventions.  The  implications 
of  standards  and  conventions  go  bqyond  the  production  of  correct  and 
reliable  software  in  their  goal  of  achieving  a consistent,  standardized 
product  which  will  be  easier  to  maintain  and  understood  by  personnel 
several  years  after  delivery. 

DOCUMENTATION  PLAN 

The  Documentation  Plan  defines  the  purpose,  scope,  usage,  content  and 
format  of  all  deliverable  documentation.  When  used  in  conjunction  with 
the  associate  contractor's  Contract  Data  Requirements  List  (CDRL),  it 
provides  direction  for  the  preparation  of  CDRL  items  as  well  as  establish- 
ing acceptance  criteria  for  appropriate  documents.  This  document,  by  virtue 
of  its  being  referenced  in  the  SOW  and  CDRL,  becomes  an  integral  part  of 
the  contract.  Its  contents  are: 


Chapter  I — Descriptions  of  documents  which  explain  the  technical 

aspects  of  software  requirements,  design  and  development, 
and  computer  listings  of  the  approved  software  routines 
and  data  structure  and  data. 

Chapter  II  — Describes  those  documents  which  relate  to  the  overall 
management  of  the  contract. 

Chapter  III  - Describes  those  documents  which  enable  the  contractor  to 

report  Items  of  Interest  to  other  agencies  and  organizations 
as  well  as  to  the  customer. 

Chapter  IV  — Describes  those  documents  which  deal  directly  with  the 
management,  execution  and  results  of  the  contractor's 
official  testing  program. 

Chcpter  V — - Presents  descriptions  of  all  documents  and  forms  which 
allow  the  contractor  to  exercise  proper  configuration 
control  of  all  the  software,  data  and  documentation. 

Appendix  A — Presents  a discussion  of  document  Issuing,  updating  and 
revision  procedures. 

Appendix  B --  Depicts  the  format  of  the  computer  usage  forecast  and 
utilization  report. 

Appendix  C --  Depicts  all  of  the  configuration  control  forms. 

MANAGEMENT  PLAN 

This  document  describes  the  managerial  approach  and  procedures  that  will  be 
followed  by  the  contractor  to  perform  the  contract  tasks.  It  covers  the 
entire  spectrum  of  activities  from  development,  validation,  to  operation 
and  maintenance. 

preliminary  design  specification 

This  document  presents  the  Implementation  concepts  at  the  subsystem  level. 

The  performance  characteristics  of  the  software  system  are  considered. 

The  components  of  the  system  described  are  as  follows: 
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Scope 

Overview  of  the  System 
Function  Design 
Data  Base  Design 
Operational  Design 
RFQ  Compliance 

PRELIMINARY  DESIGN  REVIEW 

Performance  characteristics,  changes  to  system  specification,  changes  to 
preliminary  design,  testing  plans,  and  interface  requirements  are  discussed 
and  approved  at  PDR.  All  of  these  Items  are  documented  for  future  reference 
and  required  changes  identified  as  action  items  for  resolution. 

DETAILED  DESIGN  SPECIFICATIONS  {Parts  I and  11) 

The  detailed  design  specification  describes  the  design  at  the  beginning  of 
the  coding  (build  to  - Part  1}  phase  and  at  the  end  (built  to  - Part  II). 

The  Part  I specifications  provides  a subsystem  description  which  illustrates 
the  design  and  operating  concepts.  Typically  it  covers: 

e Subsystem  Description 
e Requirements  Satisfaction 
• Design  Concepts 
e Operating  Concepts 
e Function  Description 
e Subsystem  Ihput/Output 
e Subsystem  Storage  and  Timing 
e Subsystem  Limitations 


It  also  presents  a detailed  description  of  each  function  In  the  subsystem 
as  follows: 


• Purpose 

e Description 

• Usage 

• Inputs 

• Processing 

• Outputs 

• Storage,  Timing  and  Restrictions 


Part  II  1s  an  updated  version  of  the  Part  I specifications  plus  the  actual 
listings  of  the  functions  Implementation. 


CRITICAL  DESIGN  REVIEW 


I 


involves  review  and  approval  of  the  Detailed  Design  (Part  I),  the  test  plan 
procedures,  and  any  problem  reports  and  their  resolutions.  The  problem 
reports  may  Involve  requirements,  design,  and  coding  problems. 

VALIDATION  AND  ACCEPTANCE  TEST  SPECIFICATION 

Validation  and  acceptance  testing  Is  directed  at  the  verification  of  satis 
faction  of  the  contract  baseline  requirements.  It  provides  the  testing 
strategy  and  design  to  provide  a validation  of  the  functional  operation 
of  the  software.  A typical  outline  Is  as  follows: 

• Introduction 

• Purpose 
a Scope 

• Reference 

• Testi ng  Structure 

• Test  Program  Controls 

• Development  Testing  Summary 

• V & A Testing 

• Test  Support 


USER'S  MANUAL/OPERATOR'S  MANUAL 

These  documents  provide  the  user/operator  all  of  the  Information  required 
to  use  the  software  and  operate  the  system.  They  Include  descriptive  data 
about  the  Data  Base,  deck  setups,  connands.  Inputs,  outputs,  error  messages, 
recovery  techniques,  training  and  Instructional  Information  sources,  and 
maintenance.  The  documents  are  handbook  oriented  and  become  an  operational 
tool  during  system  operation. 


INTERFACE  CONTROL  DOCUMENT 

The  purpose  of  this  document  Is  to  Identify  all  system,  subsystem,  and 
function-level  Interfaces  and  describe  all  pertinent  data  associated  with 
them.  It  formally  specifies  them  for  review  at  COR  and  for  Implementation. 
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CONFIGURATION  MANAGEMENT  PLAN 

The  primary  objective  of  this  plan  is  to  establish  and  implement  a formal 
set  of  uniform  procedures  which  will  provide  each  subsystem  developer  with 
the  maximum  latitude  for  the  independent  management  of  the  software  con- 
figuration for  which  they  are  contractually  responsible,  yet,  imposes  the 
necessary  degree  and  depth  of  control  for  ensuring  that  the  identity  and 
integrity  of  the  entire  software  system  is  maintained.  The  document 
describes  the  multiple  configuration  management  baselines,  the  critical 
events  during  the  development  timeline,  and  the  configuration  controls  and 
procedures  employed  during  software  development  and  testing. 

DATA  BASE  MANAGEMENT  PLAN 

This  plan  describes  the  roles,  responsibilities  and  schedules  necessary 
to  insure  accurate,  complete  and  timely  preparation  and  delivery  of 
COMPOOLS,  Data  Bases  and  Data  Base  User's  Guides  to  support  the  software 
system  from  initial  design  through  operation.  Included  in  this  effort 
is  the  development  of  the  Data  Definition  Specification  which  describes  the 
data  interfaces  and  structures  at  the  element  level  and  the  Data  Base 
User's  Guide  which  provides  the  user  with  descriptive  information  required 
to  use  and  change  the  Data  Base. 


PROBLEM  IDENTIFICATION  AND  CORRECTION  REPORTS 


I j Design  Problem  Report  (DPR) 

^ The  DPR  is  the  primary  means  of  documenting  problems  in  the  critical 

I design  review  material  and  transmitting  that  information  to  the  appro- 

r I 

I priate  contractor  for  corrections.  DPR's  will  be  superseded  by  the  use 

L ! of  the  SPR  upon  approval  of  the  proposed  design  documentation  by  the 

t I customer. 

! ■ 


- Software  Problem  Report 
The  SPR  will  be  utilized: 

A.  To  report  suspected  problems  in  an  existing  software  routine,  Auxil- 


i 
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B.  At  the  option  of  the  customer,  to  notify  developers  of  new  or 
proposed  design  requirement’s  and  to  authorize  the  Initiation  of 
preliminary  design  review  materials. 

C.  To  report  suspected  problems  In  the  system  software  and  associated 
documentation. 

Document  Update  Transmittal  (PUT) 

The  DUT  Is  the  primary  means  by  which  proposed  changes  to  existing  docu- 
mentation are  transmitted  for  review  and  approval  of  the  customer.  A DUT 
package  consists  of  the  DUT  form  and  change  pages  In  preliminary  form,  and 
Is  normally  Initiated  In  response  to  a DPR  or  SPR. 

COMPOOL  Change  Request  (CCR) 

The  CCR  Is  utilized  to  request,  coordinate,  and  control  all  changes  to  the 
official  COMPOOL.  The  CCR  will  be  written  In  response  to  an  SPR  that 
requires  a COMPOOL  change.  (The  CCR  will  be  referenced  on  the  MTM  initiated 
In  response  to  the  SPR. 

Data  Base  Change  Request  (DBCR) 

The  DBCR  Is  utilized  to  request,  coordinate,  and  control  all  changes  to  the 
official  data  base.  It  is  the  only  method  of  coordinating  data  base  changes 
between  the  users  of  the  data  base  and  the  agency  responsible  for  Implemen- 
tation and  maintenance.  The  DBCR  Is  also  utilized  to  close  cut  an  SPR  that 
requires  a data  base  change.  (The  DBCR  will  be  referenced  on  the  MTM 
Initiated  In  response  to  the  SPR.) 

Modification  Transmittal  Memorandum  (MTM) 

The  MTM  Is  utilized  for  the  following: 

A.  To  supply  an  explanation  Intended  to  close  an  SPR  where  no  corrections 
or  revisions  are  required. 

B.  To  transmit  changes  for  Incorporation  Into  existing  programs. 
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C.  To  transmit  a new  or  modified  routine  for  Inclusion  on  an  official 
tape. 

D.  To  Indicate  a need  for  revisions  to  documentation. 

E.  An  explanation  Indicating  that  an  accompai\y1ng  COM’OOL  change  or 
data  base  change  Is  to  close  out  an  SPR. 

TRAINING  MATERIAL 

Training  can  Include  lesson  plans,  exercises,  and  courses  of  Instruction 
covering  both  user  and  operator  aspects  of  the  system. 


i 
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Figure  B-1  Software  Products 
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MISSION 

of 

Rome  Air  Development  Center 


RADC  plans  and  conducts  research,  exploratory  and  advanced 
development  pzograics  in  coaaanJ,  control,  and  coamanications 
(C^)  activities,  and  in  the  areas  of  informatiord  sciences 
and  intelligence.  The  principal  technical  missior,  areas 
are  coimunications , electromagnetic  guidance  and  control, 
surveillance  of  ground  and  aerospace  objects,  intelligence 
data  collection  and  handling,  information  system  technology, 
ionospheric  propagation,  solid  state  sciences,  microwave 
physics  and  electronic  reliability,  maintainability  and 
compatibility . 
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